The gloves actually live in that pocket. This is my one winter coat for use with tailored clothing (I live in San Francisco, so one suffices), and keeping them there ensures that I don't have to look for them. I could store them in the other pockets, but I am, however, by nature a pragmatist, and if I put my gloves in the other pockets, then I can't easily get to my keys and phone. So, they're either on my hands or in the breast pocket.
I wouldn't say it looks affected, but certainly silly. That said, the fluffiness today resulted from not pushing them far enough into the pocket. Whatevs.
I hope, but doubt, that this has been of use to you.