It's no dark art, and I'm really not saying any thing new under the sun here.
Most men who "Eh, I don't do colors" reject out of calcified thinking really, and don't really advance in their sartorial quest. They're just happy with their buddies patting their back and feeling satisfied that they now belong to some group, when they don't really look that good at all. Maybe some do try to learn privately, I don't know.
Here's Alan Flusser, who even in his later Dressing the Man
, addressing the importance of color . . ."Color influences first impression more than any other aspect of attire."
He also gives some history with Apparel Arts' influence . . .- M