I say keep it. I have a EG 19th century oxford that is the same way. Fits great but the collar is way too tight to button. I would never wear it buttoned anyway so I don't really care. If you were planning to wear it tucked in, buttoned up and with a tie you would want to size up. If you're going to wear it casually so you'd never button the collar anyway, stick with the medium.