Right, if women were naturally 4 or 5 times as numerous as men ... but they aren't.
With polygamy, society can never grow beyond the most basic element, mother than children. They don't form a patriarch-based clan: as soon as the sperm-donor is out of the picture, the society devolves to mother-and-sons.
If you take another look at the Book of Genesis, you'll see it very clearly. There would never have been a "People of Israel" without the enslavement in Israel, because they would have broken down in a very brief period into the Sons of Leah, the Sons of Rachel, and the sons of Bilhah and Zilpah aligned with one of the legal wives. Give it a generation or two, and you have the sons of Leah and a few cadet-branch flies they swat occasionally.
Dang the autocorrect. “Mother and children.”