I used to have a pearl grey herringbone cashmere jacket. I loved it, but it is a little tricky to use for exactly the reason you highlight. I found it worked best within (relatively) monochromatic looks. For example, charcoal/mid-grey or winter-white/cream trousers & black shoes/boots or oxblood/burgundy ones respectively. Shirt white or those very pale/icy shades of blue or pink. Tie dark grey or black or with the cream trousers/oxblood shoes combo, you can go for a dark colour tie (navy, forest green, chocolate brown, etc).
I pretty much agree with both of you, in fact. For me, the default with navy suits is indeed black shoes... and most of the time, I feel that looks by far the best. But I do certainly recognise a useful/acceptable/more informal role for either very dark brown or dark burgundy. In fact, inspired by the discussion yesterday, I chose to wear a burgundy pair with a navy suit today. What I don't much like is lighter brown with navy. Obviously it has found widespread acceptability and naturally, I can rationally accept that... but my gut still tells me that it looks wrong, and I don't wear it.
If the suit is a lighter shade of blue, then lighter brown can look just fine, though. For example, a sapphire blue or RAF blue or steel blue or equally light summer suit could pair very nicely with a number of shades of brown inc. light brown. My distaste for them is predominantly with dark navy.
Anyway, that burgundy shoes/navy suit combo I mentioned. Lighting was even more terrible than usual, so the correction is equally harsh:
Beautiful pinstripe compliments ....