Actually, he sat out 14 elections (out of 16 possible) when he was registered to vote in Texas.
I used to begrudge folks for not voting, but I've changed my mind.
The chances of one vote making the difference in an election are smaller than the chances of getting struck by lighting. A single vote is statistically inconsequential. And this is even before you consider that many people vote illegally, cancelling out your vote.
So if folks have something better to do on election day, I don't blame them for not voting.
Also, there is a large number of people who don't vote because they believe that by voting they endorse a corrupt and fraudulent process. I don't blame them either. I might even wind up doing the same thing.
But you're correct about "apathy [being].... dangerous.... to freedom". I just don't necessarily equate apathy to not voting, as many people who don't vote are not apathetic and many who do vote are very apathetic.