Talk to your doctor before going off it, it may be that the side effects might still go away. If you go off of it, wean slowly. I weaned off of 40 mg in about a year (I probably could have gone faster, but I was scared). I have very few withdrawal effects, only slight dizziness here and there after dropping my dose. But talk to your doc before doing anything.
Ultimately we know deeply that the other side of every fear is a freedom. - Marilyn Ferguson
Habituation always defeats fear. - Edmund Bourne