The more global issue is that there seems to be enough doubt about the roles and role models of and for men in American, Canadian (as well as British?) society.
I rarely get this problem. Probably, ironically I know, because I never discuss clothes with people, except a select few close friends who also like clothes. With the average American man theyre happy to discuss clothes to a certain level, but if you exhibit too much knowledge or it goes to a deeper level, the relationship changes so that they either think you know too much or youre a resource for places to buy things. neither of which sits well with me.
What's funny is when I exhibit any knowledge of clothes, it's like Im "outed" and a friend's gf claimed I was "such a metro sexual", not necessarily based on seeing me well dressed but that id actually discussed what article of clothing would go with what. I think it's always been like this though. In Britain, where they make some of the best clothes, shoes, materials and jewelry for men, isnt it very "old boy" to buy nice thing sna dthen act like you just dont care?
My point is that it isnt the wearing of nice clothes that runs you into trouble, or even the preoccupation with nice clothes, it's letting others know you care, or think about them or groom yourself too much.