I have 3 pairs of penny loafers. C&J Merton and Sydney, and Cheaney Imperial Cavendish. I like them all very much, functional at my work and leisure time. Many may have different opinions about matching loafers with office attire, but I don't have any issue with it. All my penny loafers are calf leather, so I suppose that helps to an extent that I don't feel out of place and other people don't keep starring at my feet at work, unlike suede loafers which IMO are more casual. Talking about wearing loafers at work in the office, I suppose the look of the loafers does matter. I will avoid wearing thin-soled loafers (imagine loafers meant for driving purposes), because they are just too casual.
Well by now the OP already received more than 80 votes, has he made up his mind?