I'm from Los Altos. It's an interesting place. Most of the non-hills part was originally WWII era housing for people who worked at Moffett Field. There are plenty of these "tear down" houses still around and they're probably the best way to buy in to the area if you don't have a TON of money. Now, we're still talking ~$1M, but there are plenty of places in Los Altos that will be well into the $4-6M range or higher, especially if you start looking in Los Altos Hills.
There are a couple of condo complexes off Edith in/near downtown Los Altos and a couple of apartment buildings in the small triangle formed by Cuesta, El Monte, and Foothill Boulevard. Everything else is single family homes. It's probably the nicest suburban part of the peninsula. There are more affluent places, like Woodside or Atherton, but they lack the downtown and the proximity to shops that Los Altos has.
I remember what you mean about "tear down" houses. I went to Pinewood up until 6th grade. TBH I don't get to Los Altos much these days though.