But the states seeking to join, sought to do so of their own free will.
As a practical matter, yes. No territory was admitted without evidence that a majority of the people in it wanted statehood. But as Article IV declares only Congress has a say in whether a state is admitted or not. They are not required to have the approval of the people in the territory or an OK from the president. And while, as I said, the approval of the people of the territory is obtained that alone is not enough to trigger admission. Kansas and Colorado are two examples where the people of a territory requested admission and that was not done until several years later. So your free association is not entirely true. It's admission by permission. And once allowed in why is it not logical to assume that leaving would be done through the same process?