Sorry to bring it back to polo shirts but I think there are two problems with them.
The first is that with the collar up they are such an upper class no chin Henry look (both male or female) that it would make it very easy come the revolution to see who needed putting up against the wall.
That aside, I agree with the general view that they can look good if they are the right fit. However, this comes to the second problem - the tendency for their collars to end up all curly uppy after the first wear. So, a lovely idea but, I don't go there any more for fear of seeing them all forlorn in my drawers with that no longer beautifully flat collars.
If anyone knows a solution please let me know as the Mod / Skinhead look has much to commend it and Fred Perry (3 buttons) are lovely.