I only wear button downs. And if it's like a four seasons wool, sure. I have no idea on the heavier wools since I don't wear them. These probably look best with khakis.
http://24.media.tumblr.com/tumblr_mb80wn7abH1qfrjg0o1_500.jpg (excuse the trendyness, just the color) that would work just fine. For the most part, all my string loafers go with chinos and linen. I'm not sure if your office is cool or not cool with that. It might be?..