He should have just said “not leaving, go away” and got back to what he was doing.
He should have asked her to show him the written rule or provision stating that no white people could use that facility. Tell her he would leave when she showed him the official document. If she kept harassing him, call the police and report her for disorderly conduct and disturbing the peace.
The most disturbing part of the video was the very end when the two young men meekly packed up their stuff and left.
If your going to take a stand on something, stand! Otherwise, you just look defeated.