Irbe, it sounds like your loafers fit nearly the same as mine.
Started out with a little heel slip when sockless, and tight with even thin socks. They have definitely stretched out a bit, and the soles (leather in my case) have certainly broken in. Taken together, the stretch of the leather and the breaking in of the soles somewhat cancel each other out.
The following is all in regard to the current fit when sockless (with thin socks, the fit is perfect).
If I lift my foot up off the ground and shake it, the shoe stays secure on my foot with no flopping around. However, when walking, my foot does slide slightly forward on downslopes for example, and I need to make a conscious effort to push my heel back to go up stairs. Thus, heel slip is now negligible, but that is dependent on me consciously positioning my foot in the shoe to avoid it. Perhaps the current fit is not ideal, but I think that it works fine for "loafing". The loafers fit fine for strolling, but I would want a better fit for any more demanding than that. Luckily, just throwing on socks provides the ideal fit. Also, I haven't tried tongue pads yet, but they may provide a more ideal fit.
If I could do it over, I might size down in length or width, but I think that would make the shoe only appropriate for sockless wear and perhaps too tight. But I'm not sure.
Hope that helps.