Well, the Madison is a traditional cut, so if they're too slim, it seems odd that the slimmer cut Fitzgeralds would fit perfectly. But if you say so...
Have you considered Lands' End Year'rounder (yes, that's how it's spelled - one word, with an apostrophe) trousers? Although perhaps not quite up to Brooks Bros. standards, they're actually pretty good. They're reasonably priced (particularly when paired with a coupon, or when they're on sale, or both). They come in a decent variety of colors and even a pattern or two - I wore a pair in mahogany micro houndstooth, just the other day.
They come in a couple of rises. In traditional or tailored fit. And you can have them hemmed or cuffed to your preferred length, free of charge.
You could order a few pairs, in a variety of fits, lengths, colors, etc., and simply return the ones you don't like to your local Sears store, for a full, quick, and easy refund. (You don't indicate where you're located, but if it's in the US, there are Sears stores all over.)
Again, not the best pair of pants out there, regardless of price. But neither are they half bad, and they do represent considerable value for the price. And like I said, if they just don't work for you, returning them is hassle-free.
Well, just a thought. One might argue that Lands' End isn't what it once was (what is?), but it still does some things tolerably well, and relatively affordably. It's an option worth considering.