in addition to the ones mentioned...
check out Unis...I havent been there myself (the brand wasnt on my radar last time I was in NYC) but after having gotten a few shirts and trews over the past year I'm a big fan and def want to check this out
maybe rumor....but I heard that Fred Perry opened (is opening) a shop in NYC, to be followed by one in LA within a year.
If you're in SoHo you gotta go to the Paul Smith store, even if you dont like PS it's a rad store
Jean Shop store in SoHo is also worth a visit, really cool place.
also, Famous Friends near Tompkins Park is the original awesome denim shop in NYC before BiG moved in
I think Midtown has some great shops, but that depends on one's taste and preference. I dig Bergdorf, Barneys, Gant, Brooks Bros, Testoni...more city gent shit than streetwear, but nice shops for sure.