There's also a skew depending on gender (men commit more felonies), and on the state (blacks have higher proportions of felony convictions in the South). In Florida, more than 30% of black men can't vote because of felony convictions.
In addition to the falsely-convicted felons, the majority of felons were probably convicted for drug crimes which many feel should not be felony crimes in the first place. The people most affected by the laws have no legislative representation to voice concerns about them. Rapists and murders are one thing, but you can permanently lose your vote over a single marijuana plant.