Well, it's really an issue up and down the case size spectrum, right? Some pieces, you hear people say, "It wears large".
Or conversely you hear " Yeah, it's 44mm but it actually wears smaller than that."
And that can certainly be true (either way) for lots of different reasons....dial color, bezel size, the thickness of the watch, case shape, or even the strap it's worn on....chunky strap vs. thin.
Obviously physical differences in a guy's build, wrist size, not to mention how the watch is worn (casually, sleeveless vs. under a dress shirt)-- all important factors too.
Aside from that, I guess it comes down to personal taste, and maybe most importantly, just what you are used to.
Back in the day I thought Panerais were ridiculously oversized, but I eventually warmed up to the idea. Checked out the brand, the history. Tried a few on, and eventually fell in lust with the Radiomir. It doesn't seem so big to me any more. (Note: it's still a big damn watch.)
Yet I wear a Curvex to work a few times a month as well...but I have to say it's probably the smallest I could go, and obviously it's curved and rectangular. (The strap is 18mm wide). So I'm also Team Small Watch LOL.