But you said Matthew didn't know. How could he have spent so much time with Jesus and not known whether or not Jesus had biological half-siblings? He must have known. That strengthens the "until" view.
First, following Jesus in his adult ministry did not entitle Matthew to any particular insight into Mary's precise relation to the "brothers" of Jesus. Second, it was simply not Matthew's focus, in either of the passages, to convey either knowledge or speculation that he might have had.