By leaving out the second part of my comment, you mischaracterized me. I am aware of what people mean by "dress pants" and "office pants." I just happen to find the underlying notions repugnant. So, I would never call my flannel trousers "dress pants" unless speaking to someone who I wouldn't want to take the time to explain anything further to. The only thing I wouldn't wear them with is a t-shirt. Polo shirts, OCBDs, sweaters, college sweatshirts--all game. At the end, you can call them whatever they like, but I don't treat them as anything particularly special.
No one is saying they have to be special. Again, it's the formality gap. Throwing one or two more or less formal items into the whole is interesting and fun. But when the gap is that large, it just looks dumb.