My two cents, tell anyone who says "If you gotta ask then you can't" to go to hell.
You're not a character in an Ayn Rand novel who is some well spring of self-security, you're asking anonymously on the internet to just get everyone's opinions, it doesn't mean you lack "swagger".
Wear what you want to wear, as long as you do it honestly and free of pretension, then it can work. Now... you might look at pictures of yourself in a few years and kick yourself... but life is to be lived, not just spent pondering how to best live.
Finally, gotta agree with this guy below, ASCOT HATE CRIMES STATISTICS = 0
Quote:
Originally Posted by
JLibourel 
Where the hell do you guys live that men will attack guys just for wearing ascots? As I remarked before, I have never encountered the slightest unpleasantness when wearing one and have garnered many compliments (two just yesterday, both from men, and I am about 99% sure they were not gay}. Many years ago I worked in a public contact job in a pretty tough part of Los Angeles. I frequently rocked the ascot and nobody challenged me. Of course, the fact that I was a big, brawny guy probably helped.