"My rights extend only to the point where they infringe on your rights." So, for example, your right to swing your fist ends just before you make contact with my nose. If you violate my nose rights, I'll feel free to respond in kind.
Speaking only to the given example, that is not good enough for me. Prudence, a much written about legal concept, would suggest otherwise. Of course I recognize that 'prudence' is not a libertarian concept, as it is impossible to objectively ascertain its exact limitations. That to me is the number one dilemma or flaw in the libertarian philosophy. There must be room for some prudence.
I therefore state that your right to swing your fist ends just before your movement would necessitate a prudent person from taking immediate action in self defense. A threat of imminent physical harm to the person of an individual (not property), is an initiation of force.
Juries can debate as to when violence responds, with prudence, to a threat, or itself initiates it.
At the international level, we have that debate over the invasion of Iraq.
A good premise to accept - "thou shall not initiate force". We can argue over who initiates force, but great if we at least accept the premise.