I was conditioned to consider khaki cotton pants as default in school. Never been able to shake it. I also think with that khaki shades and grays are more often than not excellent matches, assuming there's enough contrast. I've worn light wool flannels with this jacket, navy as well. The jacket's texture is rough so I tend to avoid beefy, thorn-proof type trou with it. Maybe that's a mistake.
A better shot of color/texture:
EDIT: per hendrix, I would definitely be down for a nice pair of moleskin trousers in tan.
Shoreman, I agree with stitchy that grey/khaki can look great (Brunello Cucinelli does it a lot). Personally though, I would prefer a solid navy tie with that fit. Something with some texture, like a shantung or a knit. But then I'm pretty colour averse so disregard as needed ;-)
I like tweed + wool trousers in theory but rarely have occasion to wear that combo, as opposed to tweed + cotton trousers which is a classic smart daytime look. Maybe if your lifestyle involves a lot of dandified business casual it would be different.