well, for some, the escape clause is that the Holy spirit does not guide the College of Cardinals in selecting the new pope.
At that rate, if the Holy Spirit is not active in guiding the *church* then why would anyone in their right minds adhere to it?
The argument is that the ecumenical, papal affirmed body of cardinals deciding who is the valid pope is not an infallible decision, even though if it was one defining true doctrine it would be. Yet this is another issue in which RCs debate. For instance, in "Dogmatic Fact: The One Doctrine that Proves Francis Is Pope." Robert Siscoe argues,
Fr. Smith went on to explain that because the Church is an indefectible visible society, it can never adhere to a false head. There’s no need to study canon law, or spend years researching ancient Latin texts buried away in archives, to be absolutely certain that a particular pope was (or is) the true pope. All that is required to ascertain his legitimacy is to find out if he was recognized as pope by the Church. If the answer is yes, that alone provides infallible certitude of his legitimacy, as well a corresponding degree of certitude that all the conditions required for him to have become popes were satisfied — such as the condition that the papal office was vacant at the time. And the certitude of the pope’s legitimacy occurs the moment the entire Church learns of his election, provided it is not at once contested.
A dogmatic fact is one that has not been revealed, yet is so intimately connected with a doctrine of faith that without certain knowledge of the fact there can be no certain knowledge of the doctrine. For example, was the Vatican Council truly ecumenical? Was Pius IX a legitimate pope? Was the election of Pius XI valid? Such questions must be decided with certainty before decrees issued by any council or pope can be accepted as infallibly true or binding on the Church. It is evident, then, that the Church must be infallible in judging of such facts, and since the Church is infallible in believing as well as in teaching, it follows that the practically unanimous consent of the bishops and faithful in accepting a council as ecumenical, or a Roman Pontiff as legitimately elected, gives absolute and infallible certainty of the fact.
If the Church did not have infallible certitude about the legitimacy of the current and past popes, she could never be certain that a particular doctrine had been defined, or the definitive decrees of a council ratified, by a true pope or an antipope. Consequently, the object of the Faith (what Catholics must believe by faith) would be uncertain, which the devil would easily exploit to undermine the faith. The scrupulous would be paralyzed by fear, and the unstable would fall into the most outrageous conclusions. Those who denied various dogmas would only have to cast doubt upon the popes who defined them in order to justify their incredulity. This shows why the Church must have infallible certitude about the legitimacy of those she recognizes as the Roman pontiff, either past or present.
The Council of Constance formally condemned the following proposition: “If the pope is wicked, and especially if he is foreknown to damnation, then he is a devil like Judas the apostle, a thief and a son of perdition and is not the head of the holy Church Militant since he is not even a member of it.” – CONDEMNED - https://onepeterfive.com/dogmatic-fact-francis-pope (excerpts)