You are correct about “visiting” Militia. Only the governor of Virginia could give them permission to be there. In their own state (where legal), the local Sheriff can request their help. They were asked to leave, and did.
Good!
They showed good sense in leaving but not in going!