Fulton County still has the rape and kidnapping charges against him. That should be enough to hold him until the new charges are filed, although I would be surprised if they didn't have at least 3 murder warrants in hand by the time he was captured.
The feds have him on a gun charge..IIRC
You are correct about the rape charge. That case still has to be adjudicated. However, the FBI's weapons charge will hold the suspect until the rape charge is dealt with, and the other charges are filed. That's why the weapons charge was filed.
You don't have to issue a warrant if you already have the person in custody. All you have to do is file the charge against them.