ralph lauren sizes

Hello, for the standard Polo brand, what size other than the polos should I get?  According to Ralph Lauren's sizing chart, I would wear a M because I am a 32 waist(I don't know my chest size and I don't have a means of measuring it).  But I know I wear L Polo shirts and even they get a little tight around the sleeves and chest after putting them in the dryer.  So if I were to get other items like jackets or sweaters what size should I get?  I don't think jackets or sweaters are going to shrink as much as polos so should I get a M?