That’s all true. And the two patriarchs’ genetic lineage does not grant the line of Ishmael any right to abolish the inheritance of the line of Isaac.
Imagine for a moment if Abraham had taken Ishmael to the mountain. Would God have offered a ram? Would God have carried through to Isaac's descendents had Abraham offered up Ishmael? It would have been seen in God's eyes as an abomination of desolation.