The Senate instituted this system; between having to raise armies quickly during periods of invasion (best known ones being the Gallic sack of Rome, and Hannibal’s 16 year long Italian campaign) and conquering overseas territories to enrich themselves (officially the public coffers, but that was BS), large, well-trained, effective armies which didn’t require a large levy of citizens or of taxes.
The Praetorian Guard was instituted under Augustus, who also cut the regular army in half by getting rid of half of the numbered legions (reducing it to 28), integrating legions to bring all of them to full strength and formalizing the auxiliary system (and adding back approximately 28 legions in the process). The Guard not only was bodyguard for the Emperor (although as you said, that didn’t always work out), but also defended the city and acted as its only police force.
Also, the idea that there were no troops allowed in Rome prior to Julius Caesar’s Rubicon adventure is not true; Pompey played the Senate well, just it had played him after his massively successful campaigns of (1) conquest in the east and (2) defeat of the pirates, but that dance delayed his ability to respond, so he left Italy. The Senate was supporting both sides.
Trying to remember my Livy, but early on wasn’t it the intent that one council would go out with an army and the other stay at home with the other legions?
Gaius Marius was the first on to go to the capite censi (the headcount), the poor who could not afford to supply their own equipment (a requirement for the legions up to this time). He basically said "Join my legion, I'll pay the expenses and give you a wage". The army then became a profession.