There is something wrong with that--when everything you have in your wardrobe can only be paired with one thing or another. Then, you don't have a wardrobe, you have a collection of outfits. There is no room for adaptation in that setup, and being adaptable is at the heart of style.
Charcoal odd trousers are a good example of what I mean. It's not that they literally always look bad. It's that they are very rarely the optimal choice.
If this were The Guide to Sartorial Logistics, this would be a fine point, but from a taste standpoint, it's utterly off-topic. Particularly for a man who has the basics covered, there really is nothing wrong with having a shirt with limited versatility. And your objection to darker gray odd trousers is idiosyncratic. Plenty of tasteful, stylish men like and wear them.