A nice well made fedora is never out of place, sir. I commend you for your passion to wear the hat even in the face of adversity.
As somewhat of an amateur psychologist, I would suggest that the people giving you sarcastic comments are merely insecure in their own style and identity and seeing someone confident in theirs causes them a great deal of anguish. They compensate for their own securities by trying to tear you down with hurtful comments. Be a brick fortress sir.