People often forget two things concerning Scripture. It was written in ANOTHER language (in this case, Greek). The word used by the Gospel writers that has been translate as "brother" meant close relative too. Also, there are MANY ways that someone can be your brother. Literally, one can be a brother if you have the same parents, or if you only have ONE parent in common. Tradition (I know, it's not in the canon of the New Testament that was set by the Eastern and Western Churches several centuries after Christ's life) states that Joseph was probably much older than Mary when she was betrothed to him, and quite possibly, he was married once before. This is what, as someone state before, the Orthodox Churches hold (apparently). This view has certain merit, in my opinion. What Scripture appears to say due to translation doesn't give the entire picture.
Using it to vouch for a certain viewpoint that is in antithesis to another (Mary had other children vs. Mary was a "perpetual virgin") will always show bias to one or the other viewpoint. We can never be ABSOLUTELY sure, because Scripture itself isn't entirely clear on this, so as long as we don't have conclusive proof of what happened, we can't say conclusively. All we can rely on is faith, and who argues the best argument (in their each individual opinion).
I firmly believe that God would not deny other children to a woman who loved and trusted Him so much.
The religious PhD's of Jesus' dusty pathed days were great at word games, as well.
They just don't impress me.
We can play 'quote the scholars games,' too.
Those usually don't impress me much better.