As evidenced by the variations in color of african-americans, then and now, there was some mixing between owners and slaves.
Are we individually judged by the actions of our fore-bearers?
There was definitely a lot of mixing--the census in the 19th century listed individuals as "mulatto" if they seemed to be or were known to be of mixed race. Some important "black" Americans like Frederick Douglass and Booker T. Washington had white fathers. Douglass recalls being referred to as "yellow" because of his lighter complexion.
It may be true that Jefferson slept with slaves, or at least with one slave (Sally Hemings, his deceased wife's half-sister), but it still really depends on the family tradition of Sally's descendants that that happened.