I think you'd be better off focusing less on durability and quality, and more on the aesthetics of the shoe. Casualwear is a very big category and the best shoe will depend on the look you're trying to create.
When it comes to causal I have tendered toward two brands in recent years. Firstly Paraboot, I have the Michael and the Avignon but they have plenty of great styles and the quality is excellent. The second is Astoflex who do some great suede chukka boots and again excellent quality. I have a brown suede Greenflex and I don’t think you can beat them at their price point.
I agree, "casual" is very broad. I have casual shoes/boots from a number of different companies. Deciding on the style of shoe/boot and type of leather/tanning is a better place to start. Then, based on what you end up liking that will help determine manufacturers for that style/leather. Whites, Alden, Trickers, Carmina, Viberg all come to mind - but there are plenty of others.
First and foremost, I suggest deciding whether you want a true shoe or boots (chucka, chelsea, MP or jumper style).
Agree that you'll need to narrow things down a bit. In our current sartorial climate, many would consider anything made of dark leather (i.e. a loafer) as a dress shoe. Some here consider anything less than a black or dark brown cap toe oxford a casual shoe. How would you primarily dress these shoes?