There are probably few fields where books matter, mostly in the humanities, say English or History. Not in the natural sciences or social sciences.
I've lost track of who are academics besides you two, unbelraggazo and Academic2. I'm not surprised that there are so few here on Style Forum. Academics, if they dress carefully at all, seem to be OCBD, possibly a sportscoat, and chinos (that would be dressy). Very few ties; almost no pocket squares. And very badly fitting clothes. OCBD and jeans is more common.
I started dressing more carefully in part to introduce formality into my relation to students who, in the US, seem to be assuming much more familiarity with me as a working assumption. Too many "hey dude" emails! Still, I do standout on campus, if only for well-tailored clothes and a square. At this point, I'm taking it as a mission to get my colleagues to take things a bit more seriously: one occupies a certain position that I think in the end is better served by more formality in dress.