That's kind of the point. If Manton had no influence no one would be saying it but him. I'm not saying that he is the first to ever think so. Nor am I saying that the opinion that charcoal trousers are not versatile is without merit. I happen to agree. However, to say they have no history, or violate some mythical "rule" is incorrect.
I'm not sure I agree. I do agree that Manton has had a great influence on SF, but I don't think that Manton is the only reason why people are not fans of charcoal odd trousers (after all, you yourself agree that charcoal trousers are not particularly versatile). Personally, I came to dislike charcoal odd trousers when I started wearing more odd jackets and became more conscious of how trousers and jackets work together. I never had an issue with them even 5 years ago where I'd frequently be seen in odd trousers and a dress shirt without a jacket at business casual settings.
As far as history goes, I agree that I've seen a fair amount of charcoal odd trousers (of course, the fact that they existed does not mean they were seen as good or admired). As much as many of us admire certain traditions in clothing, we shouldn't delude ourselves into thinking that a majority of the population was ever well-dressed (this gets to the do they have precedent and what does that mean point). I know Manton has reasons for his navy trouser rule but don't recall him mentioning a similar rule against charcoal odd trousers. There's also always the possibility that there is / was a rule, but it is something that people no longer care about.