He's not really famous for anything he did past 1965. His early involvement in the "civil rights" movement is a huge part of his image.
I realize that fans of his music enjoy many of his later albums (Blonde on Blonde, Nashville Skyline, Blood on the Tracks, etc.), however Dylan the myth is greater than Dylan the singer/songwriter. Just ask Joni Mitchell.
His existence in the civil rights movement consists of one and one half albums. There is nothing else. His best regarded albums aren’t among those, his most famous song isn’t on one of those albums.
And to say he is some leftist hippie ignores the truth for a myth created by others not even by him. And Positively 4th Street is how he felt about those who created that myth.