Well, he was a scumbag who gravitated toward a political career. Like so many do.
He was incredibly powerful. Essentially untouchable, for decades.
Then, a crack appeared, in the form of one of his protégés, a man named Michael Boxley. Mr. Boxley was accused of rape by several women who held positions as staff members of the NY State Assembly in Albany. One of them couldn't be talked out of pressing charges, and as a result Boxley was convicted of sexual abuse, but received no jail time as a result (he does have to register as a sexual offender).
Those troubles started back in the 2001-2003 time frame.
It's interesting how Silver's trajectory began to arc inexorably downward from that point, though. In spite of his power, he was damaged goods. Eventually, corruption charges were brought against him, and he was convicted also.
Glad to hear they’re getting rid of him. Way too late, I guess.