I think the difficulty thing gets more props on here.
IRL no one cares how difficult an item is to pair. They see you and go good or not.
Purpose of dress matters. You want to look good for yourself, but how you're perceived matters as well when you're out with friends or at work, right? So if you're sartorially amazing, but look completely inappropriate for your surroundings and company then you'll get potentially unwanted attention. If that's what you were going for, then sure.
For instance, I could wear a suit to work if I wanted, but it's out of place since my bosses often don't. I can dress well without donning a suit. I'd prefer to keep that going rather than draw unwanted attention form other staff and such.