I think you're missing the point a little here, no one is calling them a polo neck at all, but Polo shirt. Theres a Huge difference.
Dont know of many Enlgish men who call trousers Slacks (and Pants too) though, I think thats very much another one of those americanisations thats crept into our language.
I'm not missing any point at all, read the whole discussion! I was simply pointing out that the name polo is used for a collar as well
As regards the word "slacks" you must be very young not to have heard it being used in England. In the 60s and 70s We had several English First World War veterans living in our house in London who called casual trousers "slacks" . My grandfather born in Ireland (1894-1969) also a First World War veteran and my father born in Ireland (1936-1989) also called casual/informal (i.e. non-suit) trousers slacks, it is not an Americanism at all, it has been in British Isles English usage since the earliest years of the 20th century. "Pants" on the other hand IS an Americanism for trousers.