Have you tried Bonobos? Bill's Khakis?
When I was having what I think are similar to your problems, I learned I was really just shopping in the wrong stores for me. I was trying to shop in J crew or brooks brothers, and while they had different baselines of fits, neither was a good choice for me, and I like both of their stuff.
I tend to keep notes of what fits/stores work for me to avoid having to relearn what has potential and what doesn't.
A bigger mistake was to do...