For purposes of this discussion, it's enough to recognize that there is a line that can be drawn.
When is a complaint and an attempt to prevent action reasonable, and when is it not?
Precisely where it gets drawn is, of course, the hard part, since the decision would most often be based on particular circumstances.
The problem with the libertarian approach is summed up in the typical libertarian retort, "if you don't like it, you can move."
The assumption underlying this is that no complaint is reasonable unless direct "coercion" (however defined) is involved. This leaves unanswered the question of what comprises "coercion," but from my discussions with libertarians at FR, it seems largely to be limited to instances or threats of physical harm.
But there are other forms of "harm" than merely physical ones. Any imposition of unwelcome costs is a form of coercion. A policy of "if you don't like it, you can move" is a de facto admission that an unwelcome cost is being imposed. The remaining question is whether the imposed cost is reasonable, or at least not overly burdensome, despite being unwelcome.
And so we come again to the question of individual vs. community interests. The question of whether or not a complaint is reasonable, ultimately boils down to a question of whether or not my actions impose unreasonable costs on others. Alternatively, the question might boil down to judging whether my actions, though they may impose costs on some, will confer greater benefits on others.
The point here, though, is that we're talking about my actions, which are voluntary on my part, and the costs I impose by taking such actions, and which are not necessarily voluntarily accepted by those on whom I impose them.
This brings us, finally, to my own primary reason for dismissing libertarians. Voluntary action cannot be separated from the idea of self-restraint, and here is where the libertarian position on things like drugs and prostitution is a useful marker.
Responsible people tend to moderate their activities out of respect for others. For example, I generally wouldn't fire up the old chainsaw at 0300, precisely because the neighbors are asleep and I don't want to bother them.
But suppose I cannot control myself, or I cater to those who cannot. Why is it up to my neighbors to "move if they don't like it?" Why should my lack of self-control be allowed to impose costs on others? The end effect of "move if your don't like it," is a society whose rules of conduct are defined by the most deviant among us.
Part of liberty is being able to define the conditions under which I live. The libertarian point of view refuses me the right to do so, unless I behave badly -- in which case others have no right to complain. That's not "liberty," it's insanity.