As of the late 80’s, it was mostly homos. Maybe you are thinking of the homos that had “adopted”? I lived just off I-5 and Carmel Valley Rd, hiked around the bluffs aboeve the beach quite a bit. No big secret who was down there.
I was there once...but my clothing optional beach of choice was Moonstone Beach in RI (now closed) and The Ledges in Vermont, which is very popular). Both were closer to my home, now I’m in a clothing optional neighborhood.