BTW, thanks for all your tips. I found that I do have a couple of dress shirts I think will work with this, including my only buttondown collar, a Faconnable purple/white candy stripe. I also thought of the AA leisure shirst or the BR equivalent that I have a few of, which would probably work well too. I'll give them a shot and see what I think.
Oh, and I hate turtlenecks, but that may be because I was forced to wear ugly cotton/poly ones as a kid. Maybe now I will find some I like. At one point I had a J. Crew bright orange wool turtleneck that I loved, but it would be too warm for this and a moth ate it anyway.
And while I'm editing the post to think out loud, 1/4 zips in a thin fitted sweater would probably work quite well too.