You need to work on your imagination since the reality is that the Founders believed impeachment would be rare particularly with judges. They were given lifetime appointments during good behavior. Making unpopular decisions was not considered "bad" behavior subject to impeachment. Making the judiciary independent was tremendously important to them and the writings on the subject make that point clear. The last thing they wanted was a judiciary dependent upon the good will of the electorate.
Initially the trial by the Senate of impeachments meant that the people had little or nothing to say in this matter since Senators were appointed not elected. Popular opinion or beliefs were excluded from having any influence in an Impeachment trial.
And it can be done to an insane judge who is nonetheless performing good behavior. And, to my mind, it ought to be done to judges who take the job of the legislature upon themselves. That's bad behavior in my book.
And who decides when they've gone too far? That's right. The House of Representatives.