Are you sure you're trying your correct size? Don't go by the J. Crew sizing chart. It is biased toward the large, in other words, you might have to size down to get a better fit. If I follow the J. Crew measurements, I'm a large but a large will fit me like a sack. A medium is almost spot on.