I don't wear Rivets (or Khakis), but many soft shoulder/unstructured sport coats go well. Rivets are easy to dress up or down. There are many, many options, but I'd suggest avoiding high sheen wool sport coats and focusing on ones with texture and those that look less suit-like. Cotton, Linen and Wool all can work. For example, from Epaulet:
Wear with Chukkas, Trainers, Monk straps,derbys or loafers . Really only oxfords would be hard to pull off, IMO. Can be done, I'm sure.