Where them with socks, always. Socklessness is a sin, an evil fad that will pass.*
Coloured chinos would work great. A suit in a less formal business environment. And there's no rule that says you have to wear brown or burgundy with a blazer and khakis. Go for a more interesting coloured sports coat, khakis, black belt....they'll be fine. And maybe go a little bolder than usual on the socks.
*I know what you think, sockless-sinners, and I don't give a fuck. It's unsightly, unhygienic and unmanly. And bad for your shoes.
I agree that socks a preferred with long pants, but what about shorts? There is no way I am wearing socks with shorts, bad for my shoes or not.
Regarding the original post, black loafers aren't the most versatile things in the world, but you can get away with them with odd jackets or no jacket. I wouldn't wear them with shorts, socks or not.