Remember now, I’m going by what the dynamics would’ve been under a 1961-69 Nixon Administration. We don’t know how popular or unpopular Brown would’ve been by 1966 (or if he might’ve been beaten by another Republican in 1962, such as Assemblyman Joe Shell, since Nixon would not have been the nominee). Had the Conservative Shell won, Reagan wouldn’t have had the Governorship to pursue until at least 1970 (unless he would’ve opted to run for Senator in 1964 instead of his friend George Murphy).
We can play fantasy games forever, but Pat Brown was doing poorly and had already promised not to run again before trying for his doomed third term.
He only won his own primary by 52% and lost to Reagan in a massive, Christine O’Donnell/Rick Santorum type landslide, Kennedy losing in 1960 wouldn’t have changed that.
Reagan owned the Governorship of California in 1966.