No, you are not the only one who has this obviously correct view. Wearing oxfords -- that are meant for suits -- with jeans just looks like the wearer is either too poor to afford a second pair of shoes or that he has beat the shit of someone wearing a suit and stolen his shoes. The impression is one or the other. Nothing else. Cops know this, of course, and are liable to pick upa jean/st. james wearer on suspicion of assault and battery, and the realy heavy charge: taste so abominable that it is an offense against nature.
Jeans go with tennis shoes, and, at their dressiest, bluchers. Wearing dress shoes with jeans is like wearing a boardroom suit to slop the pigs. You can do it, but no one in their right mind would.