Your public FTP is still SFTP, right?
Additionally all external/internet based access to the server requires a matching certificate AND secondary authentication which would include the combination of a PIN and random generated code that's good for 15 seconds.
All of our Internal access to those servers happens over a private switched network using virtual KVM's to enable console port (serial port) based access.
I developed the security requirements and control standards for our organization. They passed our own internal Risk and Audit folks as well as the Feds.
After the first of the year I'll be tightening things down further. At some point it'll make our Unix/Linux and Windows Admin's and Engineer's scream, but that's ok. My job is to protect the bank. No one gets through on my watch.