I didn’t know why so many CA Republicans turned against Hayakawa in the 1982 primary, but they nominated another RINO, Pete Wilson.
The deal breaker for many conservatives including myself was Hayakawa's vote to ratify the treaty giving away the Panama Canal. In any case, if I remember correctly, he declined to run for another term. The party establishment got behind Wilson who, if I remember correctly, didn't have any strong opposition in the primary.