Heck, no - go for it.
You can easily have different watches for different situations.
I've only got a vintage Heuer and a couple of Omegas, so I don't have a great collection, but I also have a G-Shock and a Nike watch, and they're great for mountain-biking, running and playing in the park with my kids.
Different watches for different roles. I don't wear the G-Shock with a suit, and I don't wear my vintage Heuer Carrera when I'm rolling down some fire trails as I don't want to scratch the heck out of it if I crash!