Or they could get government out of the housing business and let free enterprise work and then the population of 2040 would have adequate housing in...2040.
Spot on! They have a mix of problems in the liberal bastion that exacerbates a geographic problem. RENT CONTROL and squatters rights (homelessness). It's risky to be a landlord in that area and the motivation to add inventory to an over-regulated, very expensive, landlocked, old neighborhood is low.
The market price of housing in most of the bay area has cut the middle class out of "ownership". The middle class rents. Owners & Landlords are either VERY wealthy or are in the 3rd and 4th generation of ownership (or live in a stick dumpster). Add in government assistance and rent control and landlords can't afford to pay the insurance and maintenance for their properties. If you have a vacant property, there is a danger that homeless people will move in. And that can take months to get them out. After that, you may as well raze the site and sell the ground.
It's ironic that in a market where the investment in new housing should make folks rich is precisely where nobody can afford the risk.
(Disclaimer: I haven't lived in the Bay area since 2000. Experience could be outdated.)
There is plenty of room on the sidewalks and under bridges and in the parks etc.