For what it's worth, the vast, vast majority of Ivy League students I know got in through pure merit. The statistics support this as well. Exceedingly few have the sort of money and legacy connections necessary to get in otherwise.
It's just that even amongst such a talent pool, few have the maturity, direction, or drive to succeed as investment banking analysts. They would have more of those things if they were older, but then they would have less stamina.