Polos are brilliant casual wear, actually as casual as I will ever be seen outside of the gym.
Having entered my forties, I always wear shirts with collars, i.e. the polo has become what the t-shirt was when I was in my twenties. Dress your age, as they say.
For the office? Possibly if you work on a Saturday in July.