They were originally created for driving, and perhaps walking to one's car, but other uses have created some controversy... They are viewed by many as more casual than other loafers. Some see drivers as comparable to boat shoes; if you wouldn't wear boat shoes, you shouldn't wear drivers. I am sure there are a spectrum of other views on this forum, (some may even say drivers may only be worn with capes ;).
I see more merit in the latter view, as they are a very casual shoe; regardless, you should be wary of them, as they wear out quickly when you walk on the street, especially in the rain.