Girlfriend's comment re-Hagg fit:
"The scarf looks douchey. Everything else would be fine without the scarf."
Soft-ish collar in your correct size really helps.
If it's too tight or too stiff, obviously that will be less comfortable. Too loose is bad too, it not only looks like shit but it can cinch up if you're wearing a tie, which is also uncomfortable.