Are you sure it fits you perfectly?
My build is exact the same as you. I thought the small size chest meassure is 39", quite big though. Normally I would wear 36".
Also I used to have a Billy Reid "slim fit" shirt (it indicated so on the label), and it's still too baggy compared to other brands.
I've never seen a rtw "slim-fit" shirt that was actually slim-fit. They all still fit like sacks.