Yikes, yeah, size down then.
I've noticed pretty erratic sizing with J Crew sweaters, anywhere from a medium to an XL fits me.
I think it's the way the armholes are placed and it may be due to my chest or something but their sweaters just don't fit me. It's a shame because it's nice wool and I got them for $45 a piece. C'est la vie. I'll just stick with BB sweaters which fit me a lot better.
I think it's a structural issue so I'm just going to return them and continue my quest for clothing that fits! I have a BB sweater that fits perfectly but they don't make a lambswool sweater in staple colors.