Brilliant point! I know that both Mary and Joseph knew that he was not Jesus' biological father and we also know that, even at twelve years old, Jesus knew because, in response to his mother's query, he said he "must be about his father's business". We know he was NOT talking about Joseph's carpentry business.
OK, so you accept without question an ordinary, human use of “father” even though it’s not exact in this case, but you insist on the modern American, non-tribal (for want of a better term) use of “brothers and sisters”!