I tried it out instore and I'd say the fit is not tight and not baggy. The end of the sleeves have a somewhat tight ribbing that will grip onto your arm and hold them in place. I'm 5'8" and average build.
Their dress shirts are definitely targetted toward the slimmer type of build, but this bias isn't as strong with the polos.
I should mention that I prefer a more athletic style polo without the grippy sleeves so the Spier & Mackay style is not for me both in terms of sleeves and fabric, but I'm sure other people love it.
The polos were designed to coincide with our slim fit shirts, so they are quite fitted. They are a traditional polo in that they are made with a pique knit and have a ribbed collar and sleeve cuffs. We are really happy with the fabric. They are airy and light and soft for summer with a great natural stretch.
The ribbing on the sleeve is fitted as well, but as it is a rib, it will ease with wear as well.