Think of it as language. Many great writers have written grammatically incorrect sentences. But they do it on purpose, to achieve a particular effect. They don't just do it to prove that they will not get arrested or the world will not end if they split an infinitive (a silly grammatical rule in English).
Most writers would be well advised not to break any rules, or do so very rarely. They are only confusing themselves and their readers.
But why did they write those sentences? Because the effect was better than if they slavishly obeyed the rule. They judged things on what is important: the quality of the outcome; not on what was unimportant: the following of a rule for its own sake, or for the sake of conforming.
Following the rules only makes sense if the rules make sense. If the rules say to wear kipper ties and bell-bottom flares, or to wear short tight suit jackets that crumple around the fastened button, and leave half the bottom exposed, then following the rules will just result in a wardrobe full of ugly clothes that you never wear 5 years later when they go out of fashion. If the rules are sound then following them should never clash with what looks good. A rule that can be frequently broken to superior effect does not deserve to remain a rule, it is more honestly described as a mere suggestion.
Fashion is not like science or deer hunting. It is not based on objective performance or facts. Science is good if it makes accurate predictions about the world. Deer hunting techniques are successful if they catch lots of deer. Fashion is not necessarily good if it is fashionable, if lots of people adopt it. So, the rules in fashion deserve far less respect than the rules do in fields where they are proven to get results.
The 'rules' in style are based on universal aesthetic truths; the rules of fashion are based on short-lived popular fads and/or tradition for the sake of tradition. So the former are worth considering but the latter should be mostly ignored.
I agree about fitting in. People's reactions matter, and dress is mainly a social activity (we dress with more care in public than at home). So, I include 'appropriateness' in the rules of style. This is one way in which clothing is different to thinks like science or athletics, where being right or being best is always superior to fitting in.