A couple of observations:
I recently saw a graphic that said that clothes buying for men hits its apex in his 50s. I guess this is when men start sagging in the face and other areas and start to overcompensate by buying nice clothing (We'll all find out, right!?) Maybe you should up your target demo a bit. I would think that older men would not be into internet buying, but not sure. This graphic may have been in Put This On. I'll try to come up with it.
Streets of G'Town went out of business, I heard, and they seemed to be wanting to do what you want. I only stepped in there once so I don't know why they didn't make it. It was ok, but at the time I was not employed so couldn't splurge on their wares.
I think we are in need of some sort of huge discount mens store like Syms or Filenes basement (which are gone). I think this town has a lot of cheap ass government employees who would rather go there than Macys when they need a staple suit/shoes, for example. That would be your bread and butter. Like restaurants make all their money selling booze (They break even on the food). Then have some sort of area where high end stuff would be highlighted (for us SF folks). Filene's basement had an area like that. I think it was called "Off the Runway" or "From our flagship store" . Can't remember.
I would occasionally pull some good ties, shirts and shoes from syms and filenes.
The point about men in their fifties makes a lot of sense to me. I'm now 35, make a good income, but I have kids and need to pay for braces, college, etc. So I basically try to get pretty nice stuff on sale. When I'm 55, barring some serious problem in my career, I should be making quite a bit more money, I will be done paying for my kids, my house will be paid off or nearly so, etc. I could easily see myself spending four times as much on clothes as I do now. In particular, I could see myself spending more time in a good men's clothing store, and less time in places like Neiman's last call and Off Saks than I do now.
I would think that it would be a very good idea to make sure you include an older demographic in your plans.