The only way to make it fair would be to add more teams:
M M team
M2F team
F2M team
F F team
Of course, then would come preferential teams for tall, for short, for over-weight, for races, yada yada yada.
Short answer: it will never be fair, because life isn’t always fair.
The problem with your proposal is that trans men view themselves as women and insist that they be permitted to compete with women.
This image kind of sharpens the issue for me: