If you can stomach going to a fine restaurant and sit next to a grown man in shorts and flip flops while you are in a suit spending $300+ per person, I guess that's your choice--perhaps you don't mind eating at the table by the restrooms either... but many of us (as has been noted above) prefer to dine in an environment which is inviting to all the senses, and not just taste.
How many times have you been to a restuarant spending 300+ a person and seen a man in shorts and flip flops? In my experience once you get to a certain price point (maybe $75+ a head) even the most slovenly are shamed in to some type of standard, with rare exceptions of course.
One can be well dressed and still smelly, borish, crass, loud, obnoxious, unruly etc. Civility is more about manners than dress IMO. I'll take the well-mannered guy in shorts over the stylish lout anyday.
I just don't get why what the guy next to you is wearing bothers you that much. After a brief glance through the room don't you spend your time eating and conversing? If anything it should make you more appreciative of how awesome and urbane and civilized you are.