i received my vintage WWII peacoat today and have the same problem. The coat fits slim, its not a problem buttoning the coat but the shoulders seem too boxy for my posture and the sleeves are way too short for my arms so I will unfortunately have to let it go. Looking to get what I paid for it.
If the coat fits you in the torso, you might look into having the sleeves lengthened. There usually is an inch or more of extra material inside the cuff that can be let out. In addition to the material you see at the cuff, there is more under the lining, but not much more. The boxy shoulders you mentioned is just the way pea coats are made. For a small framed guy, they may well appear to be boxy. I have never seen it as a problem.
I have a size 40 peacoat that fits well in the torso, with no sweater, but the sleeves have always been too short. They are right at my wrists, and my watch shows. I checked a few weeks ago and found enough material at the cuff to make the sleeves plenty long. This is the original peacoat my sister's husband gave me years ago, so I would like to get it back into service (for those of you who have read the article). This peacoat is a snug fit, so a sweater isn't always needed when the temps get around freezing. The snug fit decreases the amount of air transfer around the body, allows the body to heat the air better and keeps me warmer than a larger size without a sweater. I used to frequently wear it in NYC in the winter with no sweater, and stayed plenty warm.