it was actually noted and notable at the time and discussed in the press. As in, "the brown suit, the also-ran sign of the midwestern shoe salesman, is back: the president is wearing them!!" In other words, not only did he not wear them that often, they really didn't catch on much beyond him.
I've quoted this before, but in 1993 the Federation of Merchant Tailors in London put out a press release saying "It is now OK to wear brown in town." Still, one almost never sees it in London.
You see many sartorial offences in London, but I would feel very out of place wearing country colours in London other than perhaps for a weekend walk in Hyde Park.