The slaves you mention did not belong to Grant or his wife. The were the property of Fredrick Dent. They were never allowed to travel with the Grants when they were in a state where slavery was against the law. Dent’s slaves were freed in January of 1865, when Missouri outlawed slavery. The one slave that Grant actually owned was given his freedom by Grant 1858. Plus what difference did it make whether Grant owned slaves or not. It was legal to do so time.
What difference, indeed. Yes, it was legal at the time, and it remained legal in Union states until 1865 as you point out. Thank you.