I think what Roberts is saying, in a sense, is “it’s time for the dog-track, old timer!” The sooner Stevens takes his leave, the better for the court and the country.
"time to pin a note to his sweater and drop him off at the dog track every day" No meanness meant to the elderly, but when a guy like Stevens hangs on when he's really not up to par, it's just a rotten disservice to the country and to the Constitution.