I don't see any problem with a navy wallet. For me it is an elegant color and if a see someone with a navy, well made quality wallet I think that this person has more refined taste than the average John Doe.
Navy is perfectly fine. It's not like your wallet will always be visible, and navy is not that eccentric of a color so as to shock people when you do bust out the wallet. Veterans of the board may remember a certain dark green wallet that was also nice.