Morally, that's true. But it's the job of the law to determine when our rights violate the rights of others. To argue otherwise is to argue that we don't need laws at all, other than "just make sure you don't violate anyone else's rights." Sounds beautiful, yes, but I think society might run into a bit of trouble unless its laws got a little more specific than that.
I think we have come full circle. Since you concede that one mans rights are bracketed by the next mans rights, we need to define arms more specifically, especially since the nature of arms have changed. Thusly, my argument about the bracketing of possession rights of WMD's against the rights of others to quietly enjoy their property.