I think that both would work well. They are two neutrals which will match well with many colors of either pants or shirts. The biggest difference in my opinion is that grey is more 'city' and brown projects rather 'country' vibe.
Well, in my opinion this is an overstatement. Regardless of my look, I think that such combination (especially with trousers being rather navy than blue) is a good and classic match - just like its reversal.