Light bounces off of reflective surfaces such as glass and water, and also renders colours such as blues and greens a little washed out. Polarized lenses prevent this but there are significant drawbacks:
1. Cell Phone, PDAs, Computer Screens, SatNavs and almost all other portable screens are polarized, which means that when you view them with polarized glasses they can sometimes be rendered black (And thus un-usable).
2. Omitting waves of light on a particular oscillation (polarization is essentially like looking through a Venetian blind) removes that light from your vision. If you are driving for instance, this could present problems as you wont be receiving the full spectrum of information that could be being displayed to you.
When I am flying, if another aircraft is in the distance at 3o'clock and the sun is at 10o'clock then the only way I will see the other aircraft is by seeing the sun reflecting off of it. With Polarized glasses all I see is a speck of dark in the distance which is very easy to miss. The light saved me from crashing into the guy. - Result: Never wear Polarized glasses for Flying
When I'm fishing/surfing the glare from the sun bouncing off the water is not only blinding, it stops me seeing what is in the water with any great ease. By blocking the reflected light with polarized glasses I can see through the water as the excess light is no longer visible. - Result: Wear Polarized lenses for water sports.
When it comes to driving I personally don't think that it matters, I have both Ray Ban P and Ray Ban glasses but I stick to the non-polarized ones because I can see my satnav and CD player much more easily without them.
Edit: Woah, I never realised I knew/cared that much about polarization.
This is all very helpful. Thanks.