Yep, pants.
Yeah, that's what I thought. I didn't think there's an absolute rule or formula rather than whether it looks good or not. I've paired a gray cardigan over a light blue shirt with a button down collar with slim fit khakis. Also, a tailored blazer that sits at about the waist which I think the combination looks fine.
