I don't think there's a need to have a chip on your shoulder either way on the issue.
Cream, beige or sand linen or cotton casual jacket with denim has been photo'ed by several members successfully in WAYWRN over the years (I'm think T1, TTO and a couple of others) and I think is generally moderately accepted as a casual summer outfit... provided of course that it is executed in a pleasing manner. Summer looks, being more relaxed generally, tend to have greater latitude for variety. I do think it helps massively if the jacket is relatively unstructured but a slightly structured one is OK too. I do think that wearing it with a tie is visually jarring for such an inherently casual combination, and frequently omit a square too if I do it myself. Any open-neck pale colour shirt will be fine, though. The jeans and jacket create such a strong visual contrast between them anyway that you'd typically want to go pretty easy on everything else, esp. given how casual it is an outfit.
You can also go for T-shirt underneath. If you do that, you can add stronger colour & pattern back in as a T-shirt is already casual enough to carry that without clashing. Again, T1 does this kind of fit very well. Once you get into T-shirt & denim jacketed looks, it's probably also worth browsing the SWD forum as well as this one, to broaden your sources of inspiration.