SF doesn't really have a real estate market. It's more of an antimarket. While there is a land limit issue, we're nowhere near as dense as other cities with comparable demographic attractants - there are bedroom communities of New York, LA, and Miami among others that are more dense.
SF has highly restrictive 40 foot height limits most places and the level of resistance to teardowns is high. The level of opposition one normally sees in NY to teardowns like the Hotel Penn are routinely seen in SF against razing ugly little tilt-ups, which is why ugly little tilt-ups (and their antecedents, the clapboard "Victorians" which were originally designed as low-cost, quasi-expendable row houses) dominate many neighborhoods.
The big issue is the overwhelming power of the NIMBY factions - which include interests of multiple ideological stripe. The public voice of the movement are older Prop 13 homeowners who make up most of the neighborhood associations, along with tenant groups who push the gentrification line. Behind those groups are large portfolio rental providers like Trinity Properties who want to maintain a local monopoly on entry level housing. The end result is a coalition with power comparable to that of the NRA in DC. They're basically a bunch of (pardon the pun) rent-seeking lampreys.
There has been some successful pushback. New units are being built and rent hikes are attenuating. The primary press voice of the rent-seekers, the Bay Guardian, has imploded in the last few years due to conflict between its ownership and staff. In that time it briefly and baldly evolved into a political slate card factory, but eventually the owners finally sold the paper off to the shell corporation of a Canadian conglomerate, who are consolidating them. And most of these hikes are cyclical anyway. During the last Dot Com boom in the 90's you heard similar noises about the City becoming "overcrowded," despite our having less residents than we had in the post World War II period. And back then we had many more families living here.