Of course you do. Nobody owns you. You don't even need a reason. Secede all you want.
Congratulations freeee. You have just refuted libertarianism for me.
Effectively what you have just conceded is that people have a right to associate with people they wish, which should be obvious for a libertarian, right? Actually, this is a huge problem for strict libertarianism. Case in point:
Let's build a primitive society. A thousand people. They're all living together because they agree on basic standards of conduct, speak the same language, and so on. In other words, they like associating with each other. They share a small public infrastructure, a system of laws and so on. They are living together, of course, because people cannot live in a state of nature by themselves. They need to cooperate. Libertarians would obviously agree with this, because one of the fundamental truths of capitalism is the division of labor, i.e. people specialize in what they are good at. So you have this small, homogenous, cooperative society that is surviving pretty well.
Now, let's say that two percent of the society starts engaging in homosexuality. All men. And by doing so they insult the informal, voluntary rule-structure of society. This society has deemed homosexual sodomy to be taboo because it is a) dangerous to those participating in it, and thus costs this society in terms of having some of its members sick (lowering productivity and thus making everyone work harder, kind of like homosexuals today spread the cost of their perversion over society), and b) represents a violation of the informal understanding amongst this small society that heterosexual marriage and monogamy is the norm. This behavior damages society in that way because it breaks a taboo of society, namely that sex is for procreation inside of monogamous marriage. And the 'village elders' know from experience that if that taboo is broken, then heterosexuals will start engaging in non-marital, non-monogamous sex, and cause all kinds of other problems such as adulterous relationships, children born out of wedlock and so on. Therefore, the society itself has put it's own taboo on homosexuality (voluntarily) because it is detrimental to the whole.
So, the 'village elders', along with the rest of the 'village', come to these 30 homosexual men and tell them that their behavior is negatively affecting the community. It is causing problems, and the village wants them to stop. The men say, 'no, we have a right to engage in this behavior, and we don't think we're hurting anyone'. The village elders reply, 'Well, we think you are hurting us, by hurting our society. We don't want to punish you, but we don't want you around. We don't want to freely associate with you. Either you stop doing what you're doing, or you leave the village'.
So, the homosexual men have a problem here. They don't want to stop buggering each other, but they know full well that if they continue, they will be thrown out of society. They could choose to leave the society, and go it alone, but they also know that they'd probably die because the behavior that they engage in is inherently destructive. So, they know that they can't live without society, and society can live without them and their destructive behavior that is hurting society. Society is asking them to make a voluntary choice. If they choose buggery, they get banished to the wilderness to die. If they choose to give up buggery, they get to live.
Note, society is coercing these 30 homosexuals into stopping their behavior. Their choice is either heterosexuality and life, or going out into the state of nature to continue their homosexuality and death. But, they also have a free choice. It's up to them. Society has already spoken. Under the libertarian construction of things, the society--as a voluntary association of individuals, freely making the choice to associate with one another--has a perfect, libertarian right to determine who has 'membership' in the 'club'. They've decided that homosexuals do not qualify for membership. Homosexuals desperately want in the 'club', primarily because being in the club means safety and security, but they don't want to give up their homosexual behavior.
So, on one side of the debate, we have the 'right to free association', and on the other side of the debate we have the 'right to homosexual sex'. The homosexuals can't live without the heterosexuals, but the heterosexuals can easily live without the homosexuals. And the heterosexuals don't want to be around homosexuals. And this, my dear, is where libertarianism collapses. Because just as you can assert the 'right to homosexual sex', I can just as easily assert the 'right to free association', i.e. not be affected whatsoever by homosexuals. And both are equally valid, under the libertarian construction.
So, really if the homosexuals want all the great benefits of society, they have to give up their behavior. But the heterosexuals don't need to do anything. They'll just keep on keepin' on and not think a thing of it. The way this plays out in this debate today is that homosexuals tell us they have a 'right' to engage in their behavior, which is uniformly destructive and anti-social. I would like to assert my 'right' to tell them otherwise, but unfortunately what the SCOTUS told us today is not that homosexuals have a 'right' to engage in their behavior, but that society doesn't have a 'right' to stop it. And how would we know if the village (the people of Texas) approve or disapprove of homosexual sodomy? By voting, of course, through the elected representatives of the state of Texas. Basically, all the SCOTUS did today was disenfranchise Texas voters to please the PC cops.
Libertarianism is untenable. You assert a 'right' pro, I assert a 'right' contra. And because there's no way to know who's right and who's wrong in any objective way, it all comes down to power. In my example above, society had the power to dictate to the homosexuals. In Lawrence v. Texas, a pro-homosexual SCOTUS used their power to dictate to the people of Texas on behalf of the homosexuals. In other words, the homosexuals just hired some muscle. That's it.
Cheer your 'win', libertarians. It is suitably appropriate that the libertarian version of 'liberty' only works when it's being shoved on people by autocratic judicial diktat.