There's a difference between wearing an accessory or choosing a certain type of construction and undoing buttons. Does anyone leave his shirtcuffs unbuttoned or pop open one of the placket buttons with a tie?
When we have jackets made, we have them made with breast pockets, even though they are not necessary, so that we can put pocket squares in them, which we also don't need. The problem with leaving a button on a shirt cuff undone is that it affects the fit of the shirt; a jacket sleeve's fit is not similarly affected by leaving a button or two undone.
What makes it an affectation, to my mind, is the fact that it doesn't change the visual effect of the outfit in any way, and that it doesn't follow from the normal process of getting dressed- it requires a special effort.
But the visial effect is changed, albeit subtlely. What is and is not the 'normal process of getting dressed' is a subjective matter. Tolerance for effort--and thus what constitutes a 'special effort'--varies between individuals.
Also, doesn't it strike anyone as curious that the only buttons anyone ever intentionally unbuttons, except for the collar, are the jacket sleeve buttons which until recently were found only on better makes?
The coincidence is not mysterious: it's quite plausible that many people leave their sleeve buttons undone to connote prestige or status. That doesn't mean that you or I must do it for that reason. The unknowledgeable person believes Rolex watches are the 'best'; they may be wrong, but that doesn't make a Rolex a bad watch. I like the Explorer I in particular.