I don't doubt race was a factor, but was it "quite decidedly racist"? I would say it was as much about traditional Japanese culture as the color of their skin. From what little the average American knew about it, the biggest thing was the Japanese worship of their ancestors and the Emperor. Nobody knew if that was true for Japanese born here, and it was a risk nobody wanted to take.