What I did was go down to where it was uncomfortable, i.e., it was a struggle to get that top button done and maybe even impossible, and then to go up 1-inch from there, where it was tight but still comfortable, i.e., I could get everything buttoned even though it was just slightly snug.
Then I checked to make sure the thighs were not too tight. They were too tight on the 2-inch-down sizing.
Anyway, for what it's worth... . And it is a little weird that it took me a few purchases to nail it. But I also think it would have been difficult to know exaclty for myself without going through the trial and error.