Ever heard of Justice Samuel Chase?
Associate Justice Samuel Chase was indeed impeached (though acquitted by the Senate). The only Justice who has been impeached. Article II, which deals mainly with the Executive Branch, does state: "The President, Vice President and all civil Officers of the United States, shall be removed from Office on Impeachment for, and Conviction of, Treason, Bribery, or other high Crimes and Misdemeanors."
So, I suppose, the Supreme's are "civil Officers" and can be impeached for the reasons stated. I personally would not attempt to argue failure to recuse is "Treason, Bribery, or other high Crimes and Misdemeanors" and given the virtually non-existent chanche of getting a 2/3 rds vote to convict in the Senate, it seems academic.
But I appreciate the question and the opportunity to expand my knowledge of American History. (And Jefferson doesn't exactly come off as a saint in the affair, based on my reading).