The rule book gives them authority to do so.
ML/NJ
That is one interpretation. Let me tell you this... I watched a show on the History channel about the rise of the Nazis. When they started their propaganda demonizing the Jewish people, 100,000 non-Jews were arrested, beaten or disappeared.
And their crime? Standing up and saying, "Hey wait a minute. The Jewish people are not whom you say they are." After ten years of constant propaganda, the population of Germany had no problem with the Final Solution.
So back to you... are you still saying it is OK for a referee to throw out anyone who did not threaten harm or create a hostile situation?