I agree. The pattern actually wouldn't look too bad with some nice well-worn tweeds, but only if the scarf was much longer and the tassles on the ends were much shorter (or were removed completely).
I would also suggest an old corduroy sport coat in olive or dark tan. Slightly rumpled cotton sport coat in navy should work well also.