I was conditioned to consider khaki cotton pants as default in school. Never been able to shake it. I also think with that khaki shades and grays are more often than not excellent matches, assuming there's enough contrast. I've worn light wool flannels with this jacket, navy as well. The jacket's texture is rough so I tend to avoid beefy, thorn-proof type trou with it. Maybe that's a mistake.
A better shot of color/texture:
EDIT: per hendrix, I would definitely be down for a nice pair of moleskin trousers in tan.