I think I know where you're coming from. The orthodox choice surely would have been a dark brown oxford of some kind. But, given the texture of the flannel and that the Chukkas are as dressy as Chukkas can get (very dark brown, no casual details like a light sole etc), I thought it could work.
Suede oxfords / bluchers would work IMO. You get the texture but the formality is on the same plane.
At any rate, it's nitpicking and you look good despite Hampton's monosyllabic grunting.