Aromatase inhibitors of any kind can give you permanent side effects. So it’s definitely the reason for your problems. And I’d bet you’re not the only one with these problems. There are plenty of stories of people getting side effects from Arimidex which is used to treat Gyno. They’re mostly female but same side effects if you read on. You could find males reporting side effects too if you google it. Here are some female stories though:
Edit: here are Tamoxifen (Nolvadex) stories too: https://www.askapatient.com/viewrating.asp?drug=17970&name=NOLVADEX
I have minor gyno from finasteride and I’ve avoided using any type of estrogen blockers for the reason that it could make my symptoms worse.
You can’t mess with DHT or estrogen, you’re just asking for trouble.