@broken_oracle, Sounds to me like you may want to go with the classic dress brown leather. By all accounts it is pretty hardwearing, and I think it fits well in both work and slightly dressier contexts. And you wouldn't need to line it. There are not many pics of the HH with real wear as you say, though i recall seeing one before and slightly after pic, which I will see if I can find tomorrow. My memory is that it picks up a lot of wear quickly. No doubt more durable than CXL, as you say, but equally susceptible to aesthetic wear perhaps.
Sizewise, my own experience is that White's feels a half size long compared to brannock, but more narrow than I expected, so I'd go with the E width.
Yeah, some people really dislike the Swing last, which provides extra width, because in some pics it seems a little shapeless. But there are pics from a few months back of a pair of SDs in brown dress on the Swing last, with captoes, which look really great I think -- check 'em out.