I love the complete nerdiness of what's being appropriate. It's clear that reading the situation is rather though, so let's work this out.
He will be the one with a jacket.
1. His family won't use jackets.
2. Her family won't use jackets.
3. Neither do their friends.
Obviously they are perfectly happy and comfortable without dressing up. And they are 100% right, it's their wedding. What's disrespectful in showing up in a nice jeans and tie if that's what his sister obviously prefers?
In comes the brother, who they haven't seen in a long time. Let's go stand out and go smart. So what's the message here? Look how good I dress and your friends suck? Look how sophisticated I became (which is a lie considering he needs advice here).
I am 100% sure that's not what he wants to convey, but that is what people can think... why they take that utterly unneccesary risk? What's the upside of dressing up?
Show up in nice natural clothes and be there for her. Don't try to be someone you aren't. especially when you haven't seen them in 18 months (!)
Dress for the occasion.