I may be in the minority here, but I vastly dislike their sweaters. I find the fit overly blousy and lengths/collar widths really strange. I had one of their lambswool sweaters - I mean, it seemed nice, based on my inexperienced hand (I have no idea how one would even gauge the relative quality of lambswool), but the fit threw me off.
Are you sure you're trying your correct size? Don't go by the J. Crew sizing chart. It is biased toward the large, in other words, you might have to size down to get a better fit. If I follow the J. Crew measurements, I'm a large but a large will fit me like a sack. A medium is almost spot on.