How very convenient. That said, the frequency with which one sees statements like, "if you don't like it, you can leave" would tend to indicate that this is part of "mainstream" libertarian thought, so we needn't worry about your unwillingness to deal with the extremists.
And this shoots to the major point a libertarian will raise to you; that any society naturally sees situations in which conflicts of interest arise, and that there will be a loser.
True enough -- and the means by which the conflict is resolved are what's at issue here. And that's precisely why the typical retort, "if you don't like it, you can move" (or its sibling, "nobody has a right to not be offended") becomes an important component of the argument.
Under those rules, your ONLY alternatives are to put up with my choices, or leave -- in other words, you have no choice but to accept the costs that my actions impose upon you. There is only one "loser" in that scenario, and it's never the person whose actions impose the costs.
I suspect you'd relegate this result to the "extreme" side of libertarianism -- and if so, you have conceded that sometimes my voluntary actions can be curtailed, because the costs they impose upon you are too high.
Who decides what is unreasonable? The government?
Yes, sometimes. As a conservative, I prefer that this redress be kept to a minimum, but sometimes it is appropriate for the government to take action. But regardless of who decides, the point is that somebody has to decide -- and one would certainly hope that the criteria for deciding are both non-arbitrary and enforceable; which says, really, that it's a government decision even if you don't call it that.
This teeters greatly towards the Marxist idea of "The rich/priviledged will not miss what they have, because they have so much."
And by the same token, the opposite view "teeters greatly toward" ushering a Mogadishu-style anarchy into your neighborhood. Which is really to say, "teeters greatly" is an alarmist rather than rational term. All it does is presuppose that any and all human actions take place on the razor's edge of tyranny and anarchy -- which in real life they almost never do.
The problem comes with who decides how much restraint is enough? The prevailing will of the people? If the will of the people that prevails is more permissive than you desire, does that make it wrong?
Yes, the prevailing will of the people. That's the reality of life among other people -- a fact which libertarian theory never seems to grasp. The only safeguard against extremes in either direction is as Mr. Adams said all those years ago:
"We have no government armed in power capable of contending with human passions unbridled by morality and religion. Our Constitution was made only for a religious and moral people. It is wholly inadequate for the government of any other."
Libertarianism without prior moral foundation leads to anarchy. Government action without prior moral foundation leads to tyranny.
Ironically, the libertarian ideal of "self-ownership" has the effect of undermining the religious and moral foundations of which Mr. Adams spoke. Libertarianism is its own worst enemy.