What you suggest sounds like a recall? With a recall, first the state voters essentially hold a no-confidence vote in the executive (aka usually the governor/mayor) and if the people do not want said magistrate returning, they choose from a list of alternatives. Recalls were a product of the first Progressive Era.
Great point, along with the income tax and the direct election of senators (if the original Constitution were followed on that point, we wouldn't have this mess today).