Fabrics with a nap such as wool flannel, cashmere, or camel hair lose their nap in places where the fabric rubs against other items be it other fabrics like cloth cars seats, or against itself if there is too much fabric in the seat of your pants or between your legs. Its is common to lose the nap at the edges of sleeves on sport coats where it folds and into the inner part of the jacket sleeve and sometimes at the elbows if someone constantly leans against counters or tables a lot.
Once a fabric loses its nap, I don't think there is much you can do about it. Its generally a loss of actual fibers rather than them being matted down. Someone suggested a stiffly bristled brush, but I would not do that as it could create shiny areas where you have brushed the fabric.
For the future, be sure your pants are properly tailored (so there isn't excess material rubbing against itself in the seat or inner thighs). Also check and see if your pants are 100% wool, or a wool and cashmere blend. Years ago I found wool cashmere blends never wear as well as 100% wool and they loose their nap quickly.
Also the sad truth is some people are just harder on their clothes than other people, be it do to their work routines or body type. I knew a heavy set guy with thighs that rubbed together, and therefore he was always wearing out the inner thighs on suit pants very quickly.