That's an interesting point. That's a really interesting point, in fact.Not meaning to come at you personally here, so please don't take it as such.
But I think this thread is illustrative of some of the points being made here.
JFWR is in the chinos w/ oxfords camp, and thinks that oxfords w/ jeans (in some situations) can work. Yet doesn't wear jeans.
Similarly, other vociferous posters on the thread who support oxfords w/ chinos, or jeans, have expressed generalizations that they prefer suits or don't wear denim.
I feel like the people who are generally against the look of chinos/denim w/ an oxford have much more exposure to denim/SWD in general.
I, for example, do operations for a private label denim company (I know, Chihuahua will be upset I'm neither a famous dead painter nor a wedding planner). We ship tens of thousands of jeans and chinos in 100's of sku's every week. I walk into our showrooms both in LA and NY and see what our designers are pairing with chinos and denim. Oxfords are just, not in the wheelhouse. Not even stylistically adjacent.
I also wear denim, mostly Japanese selvedge with a higher rise and thicker hand feel. I have left the Diesel abominations behind in my 20's (though I liked them quite a bit back then - live and learn).
Yeah, I am not really into the whole SW world. That isn't my life. Especially as I associate that look with sneakers, and I just really, really, really hate sneakers.
But to be clear: I also said -sometimes-. I could imagine a pretty casual oxford, like a suede brogue, which could work with jeans of the right cut and colour. I am sure there are people who could make that look work.
As for chinos, I do wear chinos (as evidenced in this thread). I wear them quite frequently, in fact, and I just don't see the stylistic problem. I look at a man wearing oxfords in chinos and go "that seems fine to me". I also tend to wear trousers and a sport coat rather than a suit, though I do like wearing suits.