Those who will get "hammered" will be limited to the folks who took on loans they couldn't handle. The demand for housing in many areas of California, Southern California in particular, is going to continue to increase because:
1. There's lots of employment here in every industry you can imagine.
2. The climate is excellent.
3. There's only so much land for homes, while foreseeable demand for it is infinite.
"Average" is a relative term. "Average" people haven't been able to afford the "average" home in Southern and Coastal California for at least 15 years, yet average people are still here and thriving, and property values are at worst leveling off, and in many cases, rising "only" two to eight percent instead of 20 percent. There are more dynamics at work in real estate than meet the eye, I think.
Your idea of "coastal" as being within a block of the water is puzzling. I grew up in a home that was maybe a mile from the water, and you could see the whole bay and ocean from our windows. Believe me, it was most definitely a "coastal" piece of property. To me, a piece of property within a block or the water (posh or not) would be waterfront property. A subtle difference in terminology. "Coastal" property would be something within a mile or two of the water, I'd think. The view is not what makes something "coastal," but rather the marine climate and proximity to the sea. A marine climate is very, very distinct from non-coastal places, and its very possible (I know because I've done it) to live less than a mile from the beach and not see the ocean for months at a time, yet you never for an instant forget that you're "coastal." It's just very, very different from being inland.
Just my two cents, as I am a native of the California coast.
At what point would you say those factors have already been priced in? When valuations are double what they were 5 years ago? 300%? 500%? 1000%? Have the fundamentals really gotten THAT much better in such a short period of time?
One could easily argue that in many areas of Southern California, illegal immigration and crime are substantially worse than a decade or two ago. Crowding and traffic have gotten obscene. The state has gone broke and is now writing long term bonds for routine infrastructure maintenance. Native-born Americans are leaving in droves. “Progressive” zoning regulations are plopping blighted low-income housing in the middle of affluent suburbs. These trends are only getting worse with time, and the liberals in charge will fight to the death to keep it that way.
There is a very real possibility that prices will fall dramatically in the near future. CA real estate was always higher than most of the nation for the reasons you listed, and will probably remain that way as long as the pros outweigh the cons. But there is scant evidence that the most recent runup in prices is sustainable once the easy money goes away.