Factory seconds are a great way to go for those that can't afford or justify paying full price. 7 out of 11 of my pairs are seconds, and I love them. I especially like seconds for casual shoes, as I expect them to get more beat up. I only buy firsts when there is a great deal, like the McAllisters for $100 off. Actually, I think all of my seconds had extra money off, too (counting closeout discounts).
I suggest you give them a try. You should have free shopping both ways, so there's no risk -- other than getting addicted.
7 of my 9 pairs are seconds, and only a couple had a noticeable flaw, though none so bad I had to send back. I too have had a great track record with AE seconds, and strongly suggest you give it a try. J