I recently came across a black velvet dinner jacket with grosgrain peak lapels, that included a pair of charcoal gray pants with grosgrain ribbon detail. All together it looked too busy IMO. I don't think you're expected to wear velvet pants. Probably just a pair of tuxedo pants from another dinner outfit would look fine. Not sure what the "rule" is, though.