I had the exact same issue with a tag a few years or so ago (also on a Brioni square, as it happens). In my case the residual adhesive was on the reverse side of the square, and was more tacky-to-touch than really sticky, so I actually just ignored it and over use through the years it gradually eroded away. Not an ideal solution perhaps, but a little residual stickiness on a small, non-visible part of the square doesn't affect the ability to puff and use it, so I didn't make too many heroic attempts to remove it.
I suppose if it's not ignorable, one could try hand-washing the silk very lightly. You'd need to immerse the whole square, to avoid watermarks (and I'd advise a very, very light detergent like washing-up liquid used for doing the dishes), and then drying it flat on a teatowel before a light press on a very low heat, through a pressing cloth. Only attempt at your own risk, however. Ignoring it - if possible - is likely to be the better option, at least it was for me.