I roll them up as best as I can and store them in a drawer. It's definitely the neater option. I used to leave them dangling on the trousers, but have come to appreciate the ritual of putting them on the trousers when I get dressed.
I only have one pair and hang them over the hook on the end of my tie rack. If you are already up to five pairs then get one of those wall mounted hooked belt racks for about $20 from The Container Store and you can hang them, display them neatly and make choosing easy each morning.