I think the trick to "pulling off" an ascot is how comfortable you are wearing one. As I said, I wear them regularly, weather permitting, and if I do get comments on them, they are mostly positive.
Also. one must make sure they are worn properly so as not to slip out of or down into the shirt. And yes, wearling them properly does meant they are worn tight enough so not to move about