Well, this is a question of degree. Basically every second you wear your shoes (no matter how much rest they've had) is "bad" for them (ignoring things like "I enjoy the way an well-worn, antique shoe looks."). So what you're really wondering is: Is a period of time after which the rate of wear increases dramatically. i.e. Most likely, the first minute of wear (assume basically an effectively inifite amount of rest prior to wear) will do less damange than the 1000th (which occurs somewhere in the 16th hour of wear) minute even though both periods of time are just 1 minute long. I imagine that there must be such a length of time because there is with all natural things (such as leather). Also, if you think about it, this is just an extention of the "Don't wear your shoes 2 days in a row" rule since for most of us something like 12 hours is a normal shoe-wearing workday, so 18 hours straight is the same as a normal day and a shorter day with zero rest in between (as opposed to the usual 36 hours of rest the alternating day method would provide)...a practice the rule would frown upon surely. So again, it comes down to figuring out if there is a length of time, after which each marginal minute of wear is causing dramatically more wear than the previous. Such a thing would be hard to measure and would obviously depend on the heaviness of use, sweating, etc. It would probably be the point where the shoe has become saturated with sweat. i.e. You keep sweating into it and since you are wearing the shoes, they don't dry very fast, so probably they are building up moisture...at some point they cannot absorb any more moisture...that's probably the critical damage point right there. I'd estimate this to be something like 14 hours, but that is an estimate with no data to back it up other than my own intuition. On the other hand, they are just shoes and meant to be enjoyed and worn (not displayed in humidity regulated, vibration controlled glass display cases with armed guards), so don't worry about it.
But it is a fun thing to analyze, is it not?