They have no real interest and would probably assume if asked that he did have a wife and children - like the Hindu gurus, the Buddha, the Sikh gurus, Mohammed etc.
Very few people as a percentage of the population actually buy Dan Brown's fake history outright.
But even if Jesus did have a family, why does it matter? This has not been proven or disproven. I saw an interesting look on the history channel at Jesus family's role in the early church and that it was not very important to church due to the role of everyone part of the same family of God after analysising the old documents that still survive from the 1st and 2nd century.