I like that cardigan. Where's it from? Wool or cotton?
I think khakis would work if they are a shade or two darker than the sweater. Olive or brown chinos/cords would probably look nice, too. I think any shirt you would wear with khakis would work. Grey or ecru OCBDs would be the safest bets, but a blue or pink solid or gingham would look good.