To say "I tried a hat and they don't look good on me" is like saying "I tried suit and they don't look good on me." Hats may be more finicky in appearance since they are on top of your head and frame the face, but I would find it hard to believe that a person simply doesn't look good in any hat. Like shoes, suit, tie, etc. would it not be a matter of finding the right hat? Color, style, fit, texture, brim, curl, crown, bash, ribbon, finish, etc. all come into play.
This isn't to imply that one has to wear a hat or even has to like or want one. But like a suit, chances are good there is one, or likely several out there that look good on you.