Would you include Mary amongst the "rabble"? Did she not call Joseph the "father" of Jesus?
Luke 2:
[48] And when they saw him they were astonished; and his mother said to him, "Son, why have you treated us so? Behold, your father and I have been looking for you anxiously."
[49] And he said to them, "How is it that you sought me? Did you not know that I must be in my Father's house?"
[50] And they did not understand the saying which he spoke to them.
[51] And he went down with them and came to Nazareth, and was obedient to them; and his mother kept all these things in her heart.
[52] And Jesus increased in wisdom and in stature, and in favor with God and man.
Brilliant point! I know that both Mary and Joseph knew that he was not Jesus' biological father and we also know that, even at twelve years old, Jesus knew because, in response to his mother's query, he said he "must be about his father's business". We know he was NOT talking about Joseph's carpentry business.