First, it depends on what fiber/weight/weave and color(s) of odd jackets you like to wear. That said, in my view, gray flannel should always be #1. This is THE odd trouser, and perfect with everything. If you can afford to go nuts, it's to have them in various weights (say, at least two) and colors (medium to light look best, I think). You might also consider a gray twill worsted. Not exciting, but it can be useful. After that, brown cavalry twill is nice, as is tan gabardine. There a lot of cotton choices. For cooler weather, moleskin and corduroy are great, and work in many, many colors. I happen to like olive for corduroy, because it looks good in itself, and I really don't like olive wool. For warm weather, light tan or cream brushed khaki and linen are nice. --- This is what I do pretty much. Why go Navy, when you can get so many wonderful shades of gray/grey all the way to gunmetal grey and fabric/patterns and weights...endless and almost always dead on with grey/gray. I also like dark brown for fall / winter, and light tan for spring/summer. An off white linen for hot summer. And evey a lux sheen linen grey/gray for summer nights.