Actually, he was right.
The 1st Amendment only prohibits government from abridging your right of free speech.
Non-government individual citizens can counter your speech with their speech.
I’d always understand that my rights ended where your rights began, and that the reverse was also true.