Mike always recommends sizing based on thighs, and adjusting waist/seat to match. Sounds like you could size down and let them out if you're really intent on a slimmer leg. IMO, if they fit well now I would leave it alone and continue as you have been. It depends if the extra alteration cost is worth it to you. Some fabrics might show external evidence of being let out?
So I've basically followed the above advice, except I don't bother letting them out, and it's close enough to not be an issue.
I echo this advice 100%. Size for dem thighs.
I am in the all-too-similar situation of having a higher thigh/butt : waist ratio. I get the 36 rivets and have the waist taken in to 34 inches, and the fit with just that alteration is the best chino I've ever worn.
In fact, I'm wearing my 7oz French Blues today, with it still in the 70s and summer not wanting to die completely yet. Noticed when pulling them on this morning that they're fading through a little bit, particularly at the knees, compared to the saturated color when they first arrived. These are going to be fantastic casual pants for summer, but a part of me does wish they were a tad more colorfast.