Migrations are often caused by bad conditions. If a group of people are migrating because of famine, a disproportionate number of the women are likely to die along the way.
Bad conditions would imply that those left behind were even more negatively selected. In fact, the far higher genetic diversity within Africa suggests that the emigrants were the marginal non-dominants being outcompeted for resources and access to mates.
Any polygamous species always has a “halo” of beta- and gamma-males who can’t challenge the alphas who exist on the margins of the main herd’s habitat.
It stands to reason that at some point a number of these betas, with what few women they had, opted out of competition with the alphas and decided just to strike out into entirely unknown territory.