Depends on how much you want to layer, but it certainly looks big. When I tried it on, the L was pretty much perfect, and the XL felt like I was swimming in it. So a size down could make a big difference. The only thing worth watching out, is that if you already feel like the L is tight in the shoulders, I can only imagine the M being unbearable. Shoulders on the L look alright, everything else looks too big and too long.
This is exactly why I didn't buy one. If it fit in the shoulder the body and arms were far to big and if it fit in the body I felt like I was bursting out of the shoulders.
Very oddly put together jacket. Or maybe I'm oddly put together. Either way, some jackets just don't fit some people.