That's probably a bad idea, if you can even pull it off. If you're absolutely dead set on the fewest colours possible, get navy and brown. Also, getting clothes that fit well will probably do a much better job of making you look 'fitter'.
I've tried it on, and I really dislike it. Looks like one of those puffy Michelin Man jackets (albeit a bit slimmer), even in real life. Can't wait for Uniqlo x Undercover! By the way, I'm selling a Uniqlo +j flannel blazer, fits a 36-38