I don't think the shirt is too casual for a tie. It's just too casual for THAT tie (although I'd never wear that tie in public anyway). Two shades lighter would be better IMO, as well.
I recently paired my navy blazer with white MOP buttons with a pair of linen trousers, like you mentioned in your OP. I like the look, personally, and if you don't mind losing some versatility it's nice. Brown horn is the safe option,...
Saw him wearing a nice district check SC on TV a while back. Odd for a TV personality, but that's just good taste. Which, btw, is not the sole domain of SF.
Oh man, I don't know if I can do next month. I will be back in the U.S. and have access to a Brooks Brothers again. I'm thinking more than a few purchases will come from it.