Dressing well can be done at different levels of formality. If you're feeling tension, its probably because your level of formality is too high. Dressing well while avoiding the "where are you going after this?" sorts of comments is the pursuit of many here.
I would look into the way different fabrics / cuts / colors / jackets impact the perceived level of formality. Its a more nuanced subject than it appears.