I wear a t-shirt to work. Your point?
My point is that wearing a jacket is socially acceptable when you're trying to look respectable. I just like to look respectable more often than not. It had worked well for me so far.
It all comes down to what you're trying to achieve. I like to be perceived as a social, well-off, young guy with a sense of style. Fitting well both into a younger and an older crowd. I like good restaurants, operas and beautiful women in elegant dresses. For me, dressing the way I do makes me fit into that world I want to be in. It seems to me that most people want to live in some version of that world, otherwise all of the things I described would not be the majority of businesses and advertisements, as well as high-class societies around the world. YMMV.
In LA it is mostly a version of that world where people wear tee shirts...