AAS, three piece tweed suits are really versatile, but since they are a rare bird I usually tone down the other elements. The shirt is fine - have you got a knit tie in burgundy or black? A white hank? Black or color 8 brogues? That might serve to mute the overall look while keeping it casual.