Not if you're going for a relatively formal city fit. For example, it's casual Friday, so you don't want to wear a suit, but you still need look formal. It's the perfect look for that, and it looks quite striking if done well. Navy blazer, white shirt, navy knit or somewhat casual tie, mid grey trousers, and well polished black loafers on a sleek last is a common look in London.
My point is that not many people on SF work in environments where suits have to be worn nearly every day, and mainly smooth worsted navy and charcoal ones with black shoes, while many people in finance and law do have to wear outfits like that. Black loafers are an excellent option in that case. Brown would look entirely out of place in a conservative environment like that.