I'd recommend small in Gant Rugger, but different shirts fit differently. There are different cuts (e.g., EZ-fit, Hugger), but the cut seems to vary more based on the material than based on the type of shirt. The Syracuse twill, for example, seems to fit pretty boxy with low armholes, whereas the madras shirts fit much slimmer. There isn't a lot of taper to them, so I usually get mine taken in by a tailor.
Also, you didn't list your chest size. I have a 40" chest, and a medium fits me well. I would assume that small = 38 and XS = 36, but I'm not 100% sure.