Generally speaking, yes. For instance, wearing a business appropriate necktie with a dinner jacket doesn't casualize the dinner jacket, it's just incongruent. I think you can categorize accessories such as shirts, shoes, and ties much more than you might think. This isn't to say that there aren't exceptions, but I think they would be relatively few.
7/29/16 at 2:51pm