A nice well made fedora is never out of place, sir. I commend you for your passion to wear the hat even in the face of adversity.
As somewhat of an amateur psychologist, I would suggest that the people giving you sarcastic comments...
You are a gentleman and a scholar sir. Quality always trumps quantity. Would you rather make love to Penelope Cruz or bang all the Pussy Cat Dolls? I rest my case