The wind problem affects all hats to some extent. Not sure if there's a viable solution there. Some hats have that fishing wire thread which you use to attach to a button but I don't think that's the way to go.
I got a couple of panamas in different colours to the standard beige straw, on sale. Check out JJ's website and do a search for Graham, Bowery, Blair and Harlem styles for a Panama in a darker colour that might not show the dirt as easily!