I could. Here is why: The Senate would argue (internally) that since it is a lifetime appointment, we will be able to wash our hands of him.
The man would be a solid right vote, but he would not be able to influence others on the court to agree with him.
Interesting points. You might have something there.