“Yet it says he was not an only child.”
No, not at all. And why would that matter to you (since you appear to reject the teaching authority of the Church)?
I guess that mentioning his brothers and sisters can be rebutted if a church has some interest in giving extra glorification to Mary and insisting on her remaining a virgin even though it would be another swipe at Joseph.
I prefer the authority of the Bible to the authority of any religion. I have trouble with a church that has folks praying to dead mortals (the saints) when Jesus gave them a direct line to Himself and the Father - it's akin to asking the janitor of a bank to put in a good word so you can get a loan. I find religion, in and of itself, to be sinful in nature and prone to developing pharisee-like hypocrites - profess to have the same God, but then look down noses at hose of "inferior" religious sects. The church has no authority - the authority rests in that which the church claims to worship.