I wear the slim fit oxford in S and its definitely not tight but it fits well everywhere. The XS flannels fit closely but it fits tighter than my oxfords after washing. The sleeves have the most drastic shrinking though. I'd get S if you prefer a looser fit with your flannels. I'm 6'1" 160lb if that helps.
Anyone try the heattech sweaters? I'm curious as to how they perform, seems gimmicky but my heattech shirts are great.