I read it again. I don’t interpret it the way you did, but that’s fine.
The reason I didn’t take them to be illegal was because he made a point of saying they were speaking English. If all they could spout was Spanish, I would assume they were illegal.