You should try plaid button-ups.
Where I am, I'm on the only guy wearing plaid, so it stands out a bit. Solid color button ups are too blue-collar mid-american, while stripes only makes you look very typically trendy.
The favorite one I have is a slim rl blue label, which is a very british tartan blue-red-green. I also have an more flannelly custom fit from their regular polo line. Re-done grunge.
A silk button-up? I think that's too gawdy to wear with anything.