I think I would consider defending myself by force if they came afetr me for some trumped up charge or audit.
They won’t do that. They can just freeze your bank account and put a hold/lien on your assets. You just go on down to the heavily guarded Gun-Free Zone Federal Building and talk to the nice IRS agent about this little misunderstanding.