And therein lies the rub. Most people in this country find racial discrimination wrong so we allow government to take away the right of assembly from those who would discriminate based on race. In doing so we allow government to dictate what type of discrimination is illegal, and shouldn't be surprised when it uses that power to criminalize a type of discrimination we do find acceptable, or necessary.
I think they have the right to assemble but just not to actually discriminate against others