It's not particularly fashionable, no. 'Course, that doesn't mean one never sees it. Because the world is full of clothing (or combinations of clothing) which are sort of terrible, but which one sees not infrequently.
As sartorial sins go, this one is less severe than many others, simply because with the sweater on, not much shirt is visible in the first place. Just the polo's collar, really. Which, by most traditional standards, isn't a good match for the sweater. Still, it'd look worse with, say, a deep v-neck sweater, which would show much more of the underlying polo shirt.
As for not wearing a button-down shirt because you don't like ironing, well, if you're going to be wearing the sweater over the shirt all day, I suppose you could wear a non-ironed shirt. I mean, if the wrinkles aren't visible, who's to know? Yeah, it'd irk me to wear a wrinkled shirt, even if nobody could see it, but that's me - it doesn't have to be you.
(And I assume you will be wearing the sweater all day long, since you mention wearing wearing an undershirt beneath your sweater. And I'm guessing/hoping that you don't consider chinos and undershirt as "business casual.") (BTW, sweater over undershirt is itself a combination which many would find less than ideal.)
Or wear a non-iron shirt. Which I still find most benefit greatly from a quick iron, but they can sometimes look more or less okay even without. Frankly, I'd be a little surprised if your shirts aren't non-iron, since these days the vast majority of button-down shirts seem to fall into that category.
'Course, practicality tends to trump fashion, in any case. If you're cold, and your choices are to either sit there and shiver, or to pull on a sweater, go with the sweater.
And much depends on the norms of dress at your office. If other men are wearing polos and crewneck sweaters, and if management doesn't seem to have any objection to the combination, well... it's still not much to my liking, but neither could I call it inappropriate for your place of work, in that case.