Colbert was modestly funny for a while. Then in 2016 every joke was about Donald Trump or Bernie Sanders. It got old very quickly.
Colbert is born and bred DC. It was only a matter of time before the rabid leftism overtook him.
Letterman was funny for a while then he did his nightly ridicule of W including clips of him stammering. But when Obama stammered and said there were 57 states nothing.
IIRC Letterman first led Leno in the ratings and then fell back behind him.