My Barbour definitely gets more wear than my North Face, but the NF can look all right in casual situations: looser jeans, t-shirt, and sneakers for the NF, slimmer jeans/khakis, button down, and aldens for the Barbour.
I've been moving toward a more trad/preppy/ivy league style myself lately. More chinos and oxford cloth as of late and less denim and hoodies/flannel. However, I've been trying to keep the trad look youthful and a bit unpredictable so that I don't end up looking 20 years older than I am, or worse,like someone trying to dress up.
If they fit snug pre-soak you should just do a quick cold soak, but bear in mind they will shrink further if you ever warm/hot wash them.
You might want to return them for a larger size and give them a hot-wash, or just get a one-wash pair.