Only libertarians can make a credible argument for this. If there’s nothing to be gained from the state by crossing the border, then yeah why not? But the left is just crazy, you can’t have a welfare state and open borders. Not possible.
The make this argument in a complete vacuum thinking people are pure economic beings basically the same as widgets. No a car does not have a worldview, a religion or a political philosophy. Those people have any when you import people you bring in all that stuff.