The trick is to adjust clothes so they fit and hang naturally-- not too tight or too loose.
Clothes are cut for typically proportioned men, so I personally often taper the pants below the knee *a bit*. There is no need to go overboard-- usually you tell your tailor "taper the bottom so that the hem is X inches". Once you find a measurement for you, you can have all your pants tapered to that measurement.
Makes sense, but many suit pants are already tapered, so they can only be tapered even more! The suits I tried on had pants that were not too tight, so they probably looked OK to others, but I think I'm not yet used to a tapered look, so to me it made it looks like my legs looked skinnier and longer. I supposed it's a getting used to thing, coming from wearing more traditional, relaxed pants which "hid" my skinny legs (at least in my mind).
Another thing I think works is having a very short break. Just to hit the top of the shoe. This makes the drape cleaner, without a lot of bunching at the bottom. Especially important I noticed with the more tapered the pant/hem opening. I found that the bunching made the leg drape look worse.
I agree with you re cotton and chinos, but what to do for casual pants? Wool is too dressy, so that is a hard one.