I don't think you can make a blanket statement since, as others have mentioned, a white shirt looks fine with a navy blazer for instance.
Part of that is because a navy blazer is about as dressy as you can get without wearing a suit and part of that is the fact that navy and white look good together.
I'd say that a white shirt will (almost) never work with an earthtone SC. Or, if it's passable, still will not look as good as a blue shirt would have. As mentioned above, anything very textured won't look good against a crisp white poplin shirt.
I think people probably focus on the formality of a white shirt to the exclusion of white's other properties / uses - white shirts generally look good in the summer when the wearer is tan(er) and the rest of the color pallette is lighter as well.