Would pick a texture that works with the texture and finish of trousers you have. Not every cloth type looks right together.
I like heavier, wintery weight flannel for trousers or suits but think it doesn't work for a jacket. Just my association with the cloth. Navy flannel can be the worst offender as a lint magnet.
More and more cloth houses have hopsack in their books and in various textures. Some more refined than others. Have switched to this over worsteds.
Use a lot...