Yes, Harry's Bloor - Adam and Sarah (not 100% sure she's still there) are the SA's I usually work with. Harry's has a small selection of overpriced AE's, plus some Canali's, Ferregamo's, etc. So so selection but yeah, overpriced.
Holt's Bloor - actually not a bad selection (no AE though), but they have most of the bigger brands and of what's left anyway, there is a sale currently on. The staff there are full commission so don't expect any help unless you look like you're buying. PM me if you want to chat in more detail since I work there.
Brooks Brothers - as mentioned, they'll have their house brand (I believe made by Peale & Co), plus I think some Alden's, not sure on that one though.
While you're there, go across the street Eastward to the (RBC?) building and there's a menswear shop with a small but good range of stuff. forgot the shops name though.
Downstairs from BB's is Robert Jones who carry EG plus I think AE's too.
Lastly, if you're under size 8 or so, Brown's short man's world on Queen West does have some AE's and a few Italian brands like Maganni(?)
David's with various locations around town does have a mix... There's that shop at the NW corner of Bay/Bloor, might be Davids. Again, overpriced but there's something to look at.
Check Holt's and Harry's LAST CALL shops
or just pay $80 and buy a greyhound ticket to NYC....
hope that's a good start.