I'm not skinny by any stretch. I'm 5' 5 1/2" or so and 150lbs on a good day and I've got somewhat broad shoulders. I don't go for the skin tight look w/ j. crew stuff, as it's impossible. I just happen to fall in the unfortunate area between small & extra small depending on the cut. The corduroy workwear jacket in small was actually a bit tight in the chest, and the medium was way too big. Also, being a bit on the shorter side, just about everything they sell is too long on me. The new pinpoint oxfords for example are very long, but the fact that the buttons are so far down make it hard for me to have the shirt shortened by a tailor. I used to be able to wear the xs in their shirts, but nowadays they are a bit too tight in the chest for me. I am in pretty good shape, when time provides i train bjj & crossfit 3-5 days a week.
Here's just a quick comparison shot to visualize what i'm talking about:
that's the chambray in small. it's still just a bit roomier than i'd like, but i'm not really willing to spend another $100 to see if xs will fit.
"favorite shirt" in small, fits pretty well but still maybe a little on the long side. the sleeves are definitely too baggy.
that's an xs ocdb i've had for about 3-4 years. the fit on it is great. sadly, that's no longer the case with their shirts.
and i can't even think about buttoning the neck on any shirts i own from them unless i want to walk around all day lightheaded from restricted blood flow to my brain.