For knits depends on the fit of the piece. Cardigans, yes definitely. Tho sometimes a slimmer chest looks good on a pullover if you have pecs as long as the shoulders aren't too tight and the body/sleeves have some length. And you're trim. No gut knit action.
Anyway I picked up the L icelandic from american rag. Fuck it, it's raglan, I'll see which looks better tts or +1. Either way one of them will be on b&s.
My 2c on F/W 12 pocket knit sizing: I ordered two in XL (navy and 'black and grey') and the navy fits significantly bigger - p2p, arm length, longer in the body. No idea if this is the norm or I got some sort of rogue size in the black and grey.