Holder is apparently counting on people forgetting that voting rights is properly a matter for the states, not the federal government--we are too accustomed to the federal government usurping power it does not rightfully have. The 14th amendment recognizes the right of states to deny the vote to male citizens 21 years of age if they have committed a crime.
So if the federal government can intervene it could only do so lawfully for female prisoners and ex-cons, and for those aged 18, 19 or 20.
In practice I think most states have mechanisms whereby felons can regain voting rights but it isn't automatic. Holder presumably wants it to be automatic to increase voter turnout. If having to get an ID is horrible because it disadvantages minorities, then having to follow steps to get your right to vote back would "disproportionately" disadvantage Holder's people.
holder seems to find felons peculiarly attractive..