You refer continually to “government” as if it were some entity that is ABOVE and MORE POWERFUL than the people! That is the exact opposite of what the founders built - government is BELOW and SUBSERVIANT to the people.
It is from the CONSENT of the governed that the government derives it’s power and failing that consent - the government ceases to exist as a legitimate entity (and then often resorts to various forms of force to remain in power).
Theoretically, each and every citizen is a sovereign entity. Each of us could declare our independence from the Nation and our state. Of course, if I were to do so, the state and the nation would have no requirement to deal with me and could even forbid the passage of myself or my goods through thier territory and forbid that others trade with me - so my independence would likely be rather short lived.
If we ALL declared individual independance - this would indeed be anarchy and the founders nightmare - something they argued AGAINST in the federalist papers as the union of the parts is better able to protect the rights of the whole!
Oh - and another example of being prevented from “leaving” - try leaving the US with your mobile property to see how much the US lets you take with you ....
As to “the vote” - the southern states did what they did with the consent of the people of the south - the north with the conesnt of the north - this in and of itself does not prove that one side or the other was correct.