In Paris Street; Rainy Day, perhaps the most prominent painting of a man holding an open umbrella, the main figure is holding the handle so that it is mostly pointing away from him. It is difficult to discern the handle direction of the other figures.
Photo of @squalortoballer
I had naturally assumed that it should point left, when holding it in my right hand, as that would present less of an obstacle if I were to run into something or someone in front of me. Also, it would show off my whangee handle more.
I spoke to my umbrella maker and he said the handle should point forward.
The handle direction is important as ideally the closure tab is hidden behind your head. Curiously, I don't see any closure tabs in Paris Street; Rainy Day. Perhaps they had not been invented at the time, or Caillebotte just didn't bother to paint them.