I wouldn't recommend sizing down if your chest is around 39/40". I was at 40 last time I measured, and if it were any tighter there would be an x when buttoned.
It comes in pretty damn wrinkled, but once you fix that it's fine. It doesn't seem to wrinkle easily. Like someone mentioned above, nice cut for swd.