We don't need Specter on the Judicial committee at all because we don't need Senators there who will apply a litmus test to nominees nor Senators who have already demonstrated exceedingly poor judgment on past nominees (Bork, who else??).
Please don't stop with just the chairmanship issue. Specter can still potentially block nominees as a member of the judicial committee, simply by refusing to vote them out of committee to the floor of the Senate. If all the Democrats join him in such a vote, they will have the majority of the committee voting against the President's nominee.
You are exactly right. Specter shouldn't be on the committee at all. We need Republicans who will stand up for their President instead of cowtowing to their Demonrat brethren like Schumer, Kennedy, et. al.