It all depends on how comfortable you feel wearing one. I work in a casual environment and I wear one to work every Friday. People compliment me on my Aloha shirts all the time.
I also wear a sport coat to work in the cooler months and while it did raise a few eyebrows and inquiries the first few times, I simply didn't care and now no one asks me why I am wearing one.
What matters is that you enjoy what you wear and don't feel self conscious/uncomfortable.