The first purpose of the EC was to appoint men of national stature, wisdom, and experience. Of equal importance, the President could assume office without political debts to pay off that superseded his oath of office.
Our Framers knew from history that factions were the death of republics. To limit the practical choice of presidential electors to one of two leaders of outright political factions guarantees recurring appointment of presidents with little loyalty to the Constitution.
The Framers' President Part I.
(BTW, James Madison wasn't the chief architect of the EC)
The Framers did some incredible things, but could not be expected to anticipate everything. Despite their prescient understanding of how important checks and balances are, they didn’t anticipate the growth of huge state and federal bureaucracies that can operate outside the bounds of elections or definitive oversight. There have to be very, very strict term limits - including limits to how many years anyone can work in government in almost any capacity. Term limits should be cumulative, including any time spent in local, state, and federal office - combined. The total cumulative term should be less than 16 years, and the total allowed in any specific office should be 8 years. No exceptions, ever.
Regarding government jobs, the limit should be 16 years, total - with the possible exception of the military and people like capital security guards, and postal workers. No one should be with the IRS for 20+ years, or the FBI, or the CIA, or any other agencies for more than a cumulative total of 16 years.