I'm sure we've all enjoyed ribbing the OP a little, but let's be fair, here. Merely kowtowing to circumstantial demands isn't something one would generally ask. To paraphrase a parental classic, if you were in a room full of plebes, and every single one of them had the bottom buttons of their jackets done up, would you do yours up too? If they all insisted that you do so, through humour or not, would you then?
We've all heard idiotic horror-stories. There was a fellow here from Trinity Dublin (IIRC) who recently admitted that, after much deliberation, he'd given up pocket-squares because they were making the wrong impression. Here's an impeccably educated guy working at a venerable academy in a city that is by no means a parochial backwater, and even he had to make a completely ludicrous sacrifice. I've had people in my circuit belligerently insist that I fasten every button on a 3/2 jacket; 'if the button's there, it should be done up'.
The key thing is to negotiate a compromise between circumstance and personal style. Merely saying 'up yours, fellas' and proceeding with excessive dress is foolhardy; falling back into schlubbery is unmanly. While there are, as others have said, things to appreciate in the OP's most recent compromise, I'm not sure it goes far enough. It's certainly less formal, I suspect that apart from the absence of a tie, it made the same unfortunate impression on his colleagues. It's still fairly high-contrast; I think it would still come across as flashy, as still others have said.
I think what you're really looking for are much more casual fabrics -- linens, coarse cottons (e.g. khakis), slubby silks, maybe some *very* muted tweeds, and so on, with very basic and conservative shirts, shoes, and accessories. You may need to keep the whisky shoes in a bag; you may not often be able to go to work with a lavender shirt without making the wrong impression.