The Civil Rights Act of 1964 ended freedom of association, but Hubert Humphrey thought it maintained freedom of association.
The problem, as I see it, is there is a great deal of difference between the govt treating classes of people differently, and private citizens doing the same. If the left was forced to abide by the rules they make for everyone else, without exception, they would rebel.