From what I've seen in with the shirts I've bought and sold lately is that the casual shirts run slightly larger. I haven't handled a dish towel though, only madras and flannels. I'm about your size (41 suit, 34" waist, 16.5" neck) and like the fit of the large better, but I prefer a slightly looser fit than a lot of people in this thread. I tried an oxford in both medium and large and found that the medium was too tight for my taste. There wasn't any pulling or anything like that that would indicate that it was too small though. I also found the armholes in the medium to be too tight for my taste; you might want to ask if you can get armhole measurements before you buy.