Patagonia is top-notch quality, though I find their fit to be somewhat baggy, at least for men's stuff. If you're looking for a fleece from them, you should know that they have their technical line of stuff (R1, R2, etc.) and their more casual line (most everything else). The technical clothing will probably be warmer for a given weight, and will last a lifetime.
There's also, Arc'teryx, which probably on par or better quality than Patagonia, and they have extremely precise construction and pay more attention to aesthetics.
In terms of build quality, almost all the major outdoor brands that you'd find in a store like REI are very well built and very durable. A step down from the two brands above are Marmot, Mountain Hardwear, and the REI and EMS house brands. They all make really good clothes, and I believe most of them have a lifetime guarantee. North Face also makes some good stuff, but they also make some not-as-good stuff, and it's generally overpriced for the quality, in my opinion. It's probably because NF is well-enough known to be a fashion brand, sort of like Nike.
All of the companies above use generally the same set of technical fabrics from companies like Polartec
You could wear fleeces from any of these brands every day for years without them falling apart. And if they did, you could generally just take them back and get them replaced.
There's also lower-end brands like Columbia and generic. They don't use the same quality fabrics, but a fleece from Columbia will still work fine and hold up well, and it'll be inexpensive.
Personally, I'd say that Marmot and the REI + EMS house brands offer the best value if you want something that's functional and durable.
