I'm thinking this is purely a rhetorical question, right? But in case you are not really sure about what I believe regarding this, let me reiterate, I believe as you probably do (if you are a Christian) that Joseph was NOT the biological father of Jesus. That the neighbors thought he was is understandable since they, at the time, did not believe that Jesus was the Messiah. I do as I hope you do, too. Jesus is Almighty God incarnate who took on human flesh by being born of the virgin, Mary, just as it was prophesied he would (Isaiah 7:14). Are we cool on that now? :o)
But you do know in other societies "brother" is still used loosely? If you know any recent Somali immigrants, for example, they use "brothers and sisters" in a more or less tribal sense. The nuclear family as the norm is a pretty recent invention.