I often get a few snickers or cracks from men, but only ever behind my back when I'm presumably out of earshot. This happens a lot in the summer when I wear my off-white Zegna blazer that has, somehow and miraculously, survived eight years of regular use without a wine or food stain. Women, on the other hand, often ask me if I'm "from Europe or something." (I once answered that I was from Japan, to which the woman said, "That's so cool; I love sushi.")
The key is just having the balls to carry it off. If you don't look self-conscious, you'll usually be fine. (Also, the best way to respond to someone who asks, "dude, why are you wearing a suit?" is to answer, "the question is, why aren't you?")