I'm not sure of the answer, but it probably depends on the age of the shoes, as well as their original retail price. Currently, of the high end American shoe companies (other than a couple of niche brands), only Alden and Allen Edmonds continue to manufacture in the U.S. However, several American shoe companies used to manufacture within the United States, including Sebago, Bass, Johnston & Murphy, Cole Haan, Sperry, and Florsheim. Of these, Sebago and Bass seem to be likely candidates (In addition to selling shoes under its own label, Sebago still does the basic, non-Alden penny loafer for Brooks Brothers, and Bass does the Weejun penny loafer (a Trad standard)). I suppose Alden is a possibility, but I've never heard of them being linked to Polo (which doesn't necessarily mean anything).
It's also possible that Polo used more than one American maker depending on the styles and price points of the shoes they were selling at the time.