IMHO loafers are like topsiders, Nantucket reds, or barn coats, and you just need to suck it up and wear the hell out of them until they are sufficiently beat up to look good casually worn.
If you want them to look new and shiny then they are relegated to use with fine merino or cashmere sweaters and trousers if that is your style.
Loafers are one of the most versatile shoes out there. You can wear them with an odd jacket, but they won't look out of place with a casual suit either. Hell, you can even wear suede loafers with denim if that's your thing.