I should say that I never wear a suit to class. I rarely wear suits at all, but for me a suit to class really WOULD be overdoing it. A sport coat with a pair of casual trousers? Not a big deal.
You know, interesting that you mention the women dressing better than the men. One of the things that motivates me to not get lazy with my appearance is that my GF is an exceptional dresser. All around town I see women who wear nice clothes, shoes, and so on, out with men who wear cargo shorts and t-shirts. I never wanted to be one of those guys who looks like a scrub, next to a woman who looks exceptional.
Oh, and I look awesome wearing a tailored jacket with a bag slung over the shoulder.
I dress deliberately like this because 1. very few professors do any more, if indeed they ever did, and it's almost radical to do so now but at the same time, it still says 'professor' to most people; and 2. I can. Academia is one of the few areas where you can wear pretty much whatever you want, with some exceptions like Business and Law schools, which tend to follow (or at least make a nod to) the norms of the professions they serve.
That matches perfectly with what I've heard from a couple of tailors in NY recently. Both of these guys owe a significant fraction of their business to making tailored suits, and neither see that part of their business growing. The exact opposite, actually. And they don't anticipate it coming back any time soon.