Yes I think if layering is important that the medium may work out better… the small would be if you only wear it over lighter button-up shirting, etc. The medium will also give you more room if you're planning on washing/rinsing; It would probably shrink just enough for it to feel right if you still have doubts.
@Apolis and others, I am looking for a fit check here. I am 5'10 150ish this is a size M, think I could do a small? I would like to be able to wear long sleeves, light sweatshirts, and sweaters underneath.
The last two pictures make it look a little large to me, not sure if the small would be better. I probably have between 4 and 6 "spare" inches in the chest.
Thanks for your help in advance.