I’ve read about cases where firewalls were bypassed by USB sticks left in bathrooms, or just left on the ground outside of entry doors or in the parking lot.
If they’re using a USB stick they have physical access, so you’re already hosed. Best case scenario there your security is so good they take the whole computer to hack it at their leisure.