They already are!
Well, although I wasn't there, my impression is that the Minuteman Project in Arizona did not encounter violence, although there were threats from the gang members across the border. But here in California, that SWARM web site makes it sound like they will physically do something when the Minutemen arrive.