what do you think when leather (low quality) hits uneven pavement? Do you really think the leather would damage the pebbles/stones?
Your rubber shoes were unfazed because rubber "gives".
By the way, the same thing would happen to "high quality" leather too.
Just because you paid a bit more (expensive) for shoes doesn't mean they will outlast your $50 pair.