Wearing it is in no way a "ballsy" decision, and I'd wear such a jacket even in conservative settings, without a second thought. Well, to the extent that a sport coat, and not a suit, would be appropriate in a conservative setting, business or otherwise.
BTW punit12, stop using the term "suit blazer" to describe it. There are suits and there are blazers. There are even orphan suit jackets which some choose to wear as sport coats. But there are no such things as "suit blazers."