The "freedom of association" is a right that emanates from the 9th amendment:
"The enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others retained by the people."
The right to "exclude" and/or "discriminate" emanates from the 9th amendment as well.
Your government, on private property, cannot enact laws to the contrary.
If Rutgers is a private university, they are free and have the liberty to exclude and discriminate to their hearts content.