Third option is to get some corduroy trousers. You can get very thin wale, light weight cotton cords that are comfortable in summer and heavier/ wider wale for cooler weather. Cords look good in darker colors where darker colored cotton khakis look like you work at Jiffy Lube type of work pants. Cords look better with a few wrinkles and wrinkle in a better way and you still look casual compared to your wool trousers.
+1, though I find cords, well too informal... I wear wool all season, it helps that I live in london for now..., might have different conclusion if I live in HK for example.