I just recently bought a J crew button up shirt. It's a small and it's a little tight around the chest area but it fits me in the length perfectly. I have yet to wash it though. Will going up to a medium be a huge difference honestly? I'm on the short side only 5'7 but I'm average built around 170 pounds. I just want the cut to be well enough to where I don't have to always tuck it in.