Pants are a heavy canvas material but I can see what you are saying. Any example of a pant you are thinking of? Are you thinking wool for texture or drape? Sport coat material is a heavy flannel. I have paired the shawl with a slouchy pull-over and it does look good. Will photograph next time I do.
re pants: I personally would choose a lighter to mid grey (heather ish) wool pant. like tropical wool for drape and texture so it flows a little...or maybe heavier but not stiff so you can wear for winter. eg, nonnative, needles etc - brands you already like - all make pants like that I'm sure. could even go cropped nonnative pants with heavy socks and those vans. for the top - I'd go for like a fuzzy mohair pullover in like navy to slightly lighter blue. but that completely changes your outfit :P - am forming it with shoes and shawl. your nonnative robe thing might work with that... dif pants tho for sure.