That SC always going to be tough. On the one hand, it's a relatively small (but not small enough to resolve to solid) / busy pattern. On the other, there isn't enough contrast between the brown and the lighter brown / tan. It's a classic example of the middling too loud for a suit / not loud enough for a casual SC.
But since you have it and are going to wear it...
A white shirt (though it does look blueish to me in the photo) is the wrong way to go. White works with either a sharp or a summery look and this is, obviously, neither of those. BD works but it really has to be blue. And not too powdery of a blue.
The pattern of the jacket is busy enough, that a simpler square would be better. White linen is too austere / dressy for this SC so I do like the suggestion of cream silk. Or, better yet, no square at all.
Tan trousers seem like a fine match but definitely throw this further into the realm of earthtones. With a navy tie, I think I might prefer light to mid grey.