Those voluntary associations would be fine if they didn’t have all kinds of laws giving them special advantages over employers—and they do.
That's my take on it. We should have the right to associate or not associate with anybody we choose, but when certain *groups* receive special privileges, protections and advantages, it's a violation of the equal protection clause, IMHO.