Hello Ryan,
I too have experienced horrible panic attacks over the last several years. The panic attacks made me so anxious about having more. I started taking Ambien to help me sleep (it worked so well), but now I have a full on dependancy for sleeping. I do however think the dependency and worry of not having medication to sleep causes even more anxiety. So I would say stop before you get too far in because it will be easier to stop the earlier on into it you are.