"I don't see any way how I can resolve this without looking like an a-hole in someone's eyes"
I suspect this may be the pressure of the occasion getting to you. You know what? It's great that we all have standards here and try to hold onto some of the strict old-school rules of formality, attire for different occasions etc - but in the real world, both in the past and now, it's really not like that unless you're at an East Coast or British 'society wedding'.
I've been to, oh, 10 or 15 weddings in the last couple of years - that time of life - from one in a fire hall where everyone wore their very best AE polo shirt and drank beer out of plastic cups to one where the groom hired out a local concert hall for the ceremony and a club (gentlemen's club, not nightclub) for the reception. And here's the takeaway: as long as the bride and groom look happy to be there and there's food, drink, music and dancing, 95% of those in attendance will have a great time and not give a damn what anyone's wearing or how anyone behaves. And even the bitchiest 5% will concentrate on the women's outfits. The tux / suit / sport coat / worse ratio among men in attendance was pretty varied at each event, and there were certainly ones where the groom plus some random attendees were in tuxes, and even some where the groom was in a suit and some guest or the MC or the best man was in a tux. And you know what? No-one gave a toss.
Just trying to send you a positive message here - don't sweat it. Show up, smile, have a good time, pour lots of drinks. Anyone who cares about you will be happy to see you happy, and anyone who's petty enough to think you're an a**hole because your brother-in-law's wearing a tux is not worth caring about. And for whatever some idiot from the internet's advice is worth, I would not bring it up with your brother-in-law; I can't picture any way that conversation works out well. Even if he defers to your wishes out of respect for the groom, he's going to wonder why you're being so priggish about it, I think.
(Some of the 'background' pages on the black tie guide have some neat pictures from way back showing that, below the 'society wedding' level, strict dress codes have rarely been observed at weddings for decades. Guys have been getting married at 2pm in tuxes since at least the 1940s, for instance, it's not some modern degradation.)