I've felt that a lot of times too. I was always afraid I'd be creeping a guy out and worried he'd be picking up "gay vibes". Funny thing is, when you visibly get uncomfortable it makes other people uncomfortable, just not for the reason you're worried about. Unfortunately, that usually makes you feel validated in your false projection.
I once heard a comedian named Maria Bamford, who had intrusive and disturbing sexual/violent thoughts since she was a child, say that the people obsessed and terrified of being violent were the least likely to be. She compared it to how a person who feels filthy and is obsessed with washing their hands 100 times a day is the least likely to be dirty.