I have narrow, sloping shoulders, a 39-40 inch chest, 34 inch waist.
I used to think of myself as a medium, but either I have shrunk or my sense of fit has grown more acute, or clothing made for the US market has grown in size, or all of the above. So now I'm often a small.
A lot of what I wear in shirts is what I would call semi-casual, in that I wouldn't often wear it with a tie or with a coat and tie, and oftentimes the collars lack a provision for collar stays.
Some brands I like are
APC
agnes b
Barneys Co-op house label
Victor Victoria
H&M
even the Gap can do a decent shirt in some basic solid fabrics -- twills, etc. -- if I take a size small.
Among more traditional dress shirts, T&A is sometimes slim, and the shoulders are not too wide.