I have the exact same problem. Ended up going with M, which is flaring out at the waist aka I can grab on the nips of fabric on the side. Everything else fits loose, but slim in the right places. I'm hoping a hot cycle in the dryer will shrink it a bit and it will be perfect.
I found my S shirts needed three top buttons undone for it not to pull