Yes, but then comes along the 14th amendment, whereby:
“No State shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States; nor shall any State deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.”
But you are correct, the governor is an idiot but elected by Liberals - says it all. But remember, sheep will eventually bite you if they are starving.
Further discussion on this topic:
https://blog.harvardlawreview.org/contagion-and-the-right-to-travel/
This is not a slam dunk on either side imho, but of course the NJ governor appeared totally unaware of the issues and the debate.