I am not quite understanding this. What kind of inserts did you get? If you are able to fit a full length insert, it usually means the shoe is sized too big. And yet, the baffling point you added is that after removing them, they fit well. I would've suspected that it would be too loose, unless your idea of comfort is a slightly loose shoe, this may indeed be the case for those with one foot larger than another.
I can only hypothesize that perhaps the breaking in of the sole and upper might have contributed to the increased comfort and this may have been hastened because your shoes with the inserts inside must have fit really snug.
Why do you remove the inserts? Perhaps a reasonable comparison might be to try to break into the shoe without one.