@Academic2 @TweedyProf @heldentenor I know I'm missing a few more obvious ones
You mentioned a gradual shift; it seems like a good idea. I'd start with OCBD (oxford cloth button downs). The cloth is nice and casual (somewhat rough), and the collar type (button down) looks good when worn without a tie. Get 2-3 of these from places like Proper Cloth and Brooks Brothers, get a feel for them, and then...