IF it looks okay when you look in the mirror, go for it. I hate the feeling of my uncovered arms in the sleeve of a jacket but I'm not completely opposed to it.
You could also wear summer weight long sleeve shirts. Wear summer weight jackets (very light wool, linen, unlined cotton). Ditch the jacket if not wearing a tie.