I suspect less yellow in the cream trousers might help? White would be perfect. Have you tried tan.
On blue trousers, I understand the challenge but I am imagining a lightly textured slate blue wool-linen-maybe silk mix (I have such a pair). The slate blue pulls in the gray from the jacket as well as distance it from navy, and the textures would pull it away from the orphan look. You'd have to find a pair and see if it actually works.
There's a picture of Alan See from the Armoury in grey with navy trousers that I thought looked good. Can't find it tho.
Fit issues aside (on which I agree better pics are needed), the worry is that your jacket is an orphan and you'll run into the difficulties of pairing a grey lightly patterned jacket. Mr. Six's pattern is much stronger to identify it as a SC and it has other elements to distinguish it as such (patch pockets), though I could imagine some might think it still not strongly patterned enough (@UrbanComposition has a robustly patterned gray jacket that works well, recently posted on his Instagram).