I do this too. Linen sleeves rumple up and end up looking shorter. Cotton less so, but still more than tweed or flannel IMHO (then you have to account for differences between different fabric weights).Edit. AAS, keep the hat as is. Looks great on you, but I'd love to see it in one of your more subdued fits.