The data doesn't support that view. According to Marshal Cohen the chief analyst of the NPD Group, the percentage has grown from only 25 percent of men's apparel being bought by men in 1985, to 65 percent of men's apparel being bought by men in 2012.
NY Times “Gay or Straight? Hard to Tell” http://www.nytimes.com/2005/06/19/fashion/sundaystyles/gay-or-straight-hard-to-tell.html?_r=0
NPD Group “The NPD Group Reports Men’s Fashion Goes More Casual” https://www.npd.com/wps/portal/npd/us/news/press-releases/the-npd-group-reports-mens-fashion-goes-more-casual/