I think that is a valid point. For one of their own "Church Fathers" to say that, "the streets of hell are paved with the skulls of bishops", admits a peculiar truth that there can really be no sure infallibility of the magesterium. The "gift" of infallibility is nowhere mentioned in Scripture and, since they are the only ones who claim they alone have this gift, it begs the question if even that statement can be infallible.
Seems to me that the only thing worse than making a mistake yourself is believing someone else’s mistake unquestioningly.
My responsibility is to be sensitive to the correction of the Holy Spirit in my own spirit. To assume that the Holy Spirit can’t do that in me but wants me to rely on His ability to do it in someone else is a huge mistake, IMO.
Seems to me that that’s how cults are formed and perpetuated.