Sure thing. Pairing polo or rugby shirts with sport coats or blazers I think only works (1) with very casual jackets (yours is too constructed and clean, and also sports a pocket square, further upping its formality), and (2) on the proviso that the shirt and the jacket be kindred spirits (your rugby shirt is, IMO, too summery in spirit to go with the autumnal tweed coat). Here's an example of a successful blazer and polo shirt combo:
(I would have ditched the hanky, but oh well ...)
Regarding the colors, while I don't dislike them individually, I think they are too uniform and too low-contrast, lending the outfit a drab cast.