Awkward Spectrum Question

I am frequently being asked in my professional and personal life by many people if I am on the spectrum. Undiagnosed. I do show traits, I know this for a fact. I do suspect I may have it. I have chosen not to persue a formal diagnosis at this time. I'm just trying to work out if it is rude and not a question someone should ask another person. I wonder if I am being unreasoble by thinking it is a rude question. Why would you ask someone that? Usually it's to point out that what I have done in that moment reflects it, like not coping with change or some other trait or another. It's been like this for years now and I've had bullies call me Autistic because I have struggled socially as well. I could have a negative connotation to the question because of being bullied about it before. I am diagnosed with Anxiety and Depression and have for years now, I've heard traits from those can look similar. I don't know, I just needed to vent about it somewhere.