Your right to swing your fists ends at the tip of the other man’s nose.
That would be a bit inside of one’s personal space, in my opinion. A swing that close justifies a closer swing of a heavier magnitude.