Despite what I said regarding yarn technicalities, the vast majority of my own odd trousers are actually non-worsted and a smaller majority isn't even wool. Most of my winter stuff is either flannel or cotton cord, though I do have a donegal pair which I'm pretty certain is worsted. In summer, most are cotton or linen. I just think they tend to look nicer. In fact, the only worsted wool odd pair I have for summer are a lightweight charcoal pair and that's compensated for by having a faint plaid check to it to add some interest. And that one's a smooth, plain weave rather than anything textured.
I think - various worsted tweeds/donegals/etc aside - there are few worsted wool trousers most people would want or need, for winter at least, esp. when you add in the fact that most odd trousers are naturally solid colour rather than patterned. And many of those tweeds/etc you might want as a suit rather than just odd trousers. There's more utility for summer lightweight worsteds I think, if you need a slightly more formal wardrobe than linen/cottons would lend themselves too, but there I'd pick plain weaves rather something particularly textured anyway.
YMMV, of course.
I mostly agree with this. Outside of summer, I am probably wearing some sort of grey flannel trouser most of the time I am not in a suit, with the occasional cotton moleskin, corduroy or perhaps tan wool cavalry twill (I believe these may be worsteds, though). Generally, most worsteds are not things I would want as odd trousers.
In summer, lightweight wool is needed for situations where cotton and linen are too casual. Either a tropical weight wool or some sort of open-weave wool (e.g. Fresco) will do nicely.
I generally agree with all of this as well, but I do think some of the textured worsted options on Luxire's site look like they would make good odd trousers, specifically the Dugdale hopsacks and herringbones. I know mid grey flannel is typically the odd trouser of choice for just about any situation, but I like trying to find new ways to break up the monotony now and again.