That quote was from the Lincoln - Douglas debates btw. :-)
At the time there was likely hardly any white person who did not agree with this, including the abolitionists. Equality was not even an issue until after the war, and physical and intellectual equality was not an issue until well into the twentieth century.
The issue was the spread of human slavery, and then became the end of slavery where it existed.
Lincoln was properly concerned with his oath to ‘preserve and protect’ the Constitution, and this had become understood, after Andrew Jackson, to include preserving the federal union.