Yeah, I would suggest sizing down with their shirts. I ordered a 15 in their Madison cut and it was pretty huge, had to return it because the collar was so big. I wear a small in pretty much every label I've tried, and it's a fairly long list.
I own HY and Epaulet Walts/Rudys, the HY pants land in between the Walt and Rudy in my experience. Not as slim as the Walts and slightly slimmer than the Rudy. Both have fantastic fabrics and construction, either way you can't lose.