really depends where in Manhattan. The avenues tend to be loud all around, neighborhoods quiet(er), Midtown awful all the time. I've spent some late nights in Tribeca, and it's surprisingly quiet, and in a good way.
People who only visit on occasion tend to think of Midtown and Times Square as "NYC!!!!" But nobody fucking lives there, it's too insufferable with crowds, noise, and big buildings.