Im the same in both sweaters and shirts. Im mostly 48, sometimes 50 and size 5 for shirts is perfect. I tried 6 once and i was floating in it.
I can understand sizing down 1 on sweaters because shoulders are usually wide. But then the length and pits would be a concern for me