I wear a fedora in bad weather in winter, but it's had the daylights beat out of it, so at least it doesn't look I just brought it out for a costume party. My girlfriend hates it.
I tend to think fedoras are less conspicuous in a cold drizzle, say, than on a balmy spring day. For guys under 50 or so, they need the right setting, and they need a certain attire, either very dressy or a little eccentric. (My stuff leans toward the latter category.) The hat and T-shirt combo so popular with fat guys ought to be eradicated.
In summer, I wear a panama, which my girlfriend considers equally crazy. But I think she finds it more appealing than the beat-up fedora.