I have a couple full Harris Tweed suits, including one from the Southwick MTO program from this past fall. I'd say Detroit pretty solidly half-way between Philly and Duluth (both in terms of geography and climate). I've worn the suits comfortably once the temps drop into 50s, so they do get quite a bit of wear in the fall. The trousers always seem heavier, probably thanks to the lining so if the temps are about 60º or above, I'll still wear the jacket but not the trousers. One good thing about most tweeds though is that they are a pretty open weave and really breathe pretty well.
As for cashmere pants... I'd definitely be wary of anything 100% cashmere. I cannot imagine they would last too long. However, most everything I've ever seen is a mix of cotton and cashmere or wool and cashmere and that should be fine.