1 vote from the White House!
Well, Jackson was dead by then, so that wasn’t an issue. I don’t know if I can come up with a definitive “worst” of the 19th century. Pierce was equally a pitiable figure, and between dealing with the horrible loss of his son and his wife’s jet-black depression over it (and that he was plastered for much of his term), I tend to cut him some slack.
Jackson himself might be up for worst, because he was too temperamentally unsuited to serve (and his wife died right after the election from the stress of the campaign). His actions also helped in de-unifying the South politically (in the post-Era of Good Feelings) and that of the Democrat Party after the collapse of the Federalists. Andrew Johnson also didn’t handle things well, either (but the impeachment proceedings represented an overreach). Grant’s administration was corrupt, and I’m not sure his serving as President was a good idea (Ohio’s John Sherman was probably a better bet, and he had the experience and problem-solving abilities, but he lost at the conventions).
1 vote from the White House!