The Freedom to associate precludes the Freedom not to associate. The SCOTUS has already ruled this.
That's why we should amend the Bill of Rights to include it.