Today I was walking to the elevator to go down to the parking garage when there was a man already waiting who'd alreadypressed the button to go up. The elevator was on the 2nd flr on its way down and I pushed the down button. We both got on and the elevator continued going down and I got off leaving the man to ride the elevator back up. He was clearly a little peeved that he had to go my way even though he was in line first. Should I have let him go up first before pushing the button to go down? He obviously thought so. What is the correct elevator etiquette?