It'll happen. And sooner than you think.
The folks at google have undoubtedly thought of all of the potential gotchas, legal and otherwise. Production cars will probably contain redundant systems that will push reliability off the charts.
I think it would need it’s own roads.
You raise a good point, and I believe you are correct. The one issue I see here is typical humanity - people let their cars go without maintenance until they are too dangerous too drive. I doubt the automated systems will detect loose or broken sway bar links, worn tires, loose ball joints, worn tie rod ends, and so forth. Or maybe they will and prevent the car from being used, but if so, people will be furious that they actually have to fix their cars instead of driving their dangerous sh!t boxes on same roads as me.