I agree with you, he should have been removed from the case based on obvious conflicts of interest.
Unfortunately, recusal motions are filed with the judge sought to be recused, and are reviewed by other judges using deferential review standards.
To put it bluntly, that sucks.