Big issue is density. Agriculture can support, in most locations, a population 10x, 100x or more that of hunting/gathering. Look at the populations of major American Indian tribes in 1700 and the absolutely immense territories they, sort of, controlled.
In the long run, this means the farmers will win.
Just finished Guns, Germs and Steel. An interesting book, though Mr. Diamond had an obvious “anti-white” axe to grind.
When he spoke of “white” expansion into Australia and the Americas, he use words with negative and judgmental connotations like “invasion” and “conquest.”
When he spoke of similar expansion by non-white groups, such as the Bantus into most of sub-Saharan Africa and the Austronesians into Indonesia, he use neutral, objective language such as “expansion” and “spread.”
It is likely the groups displaced had similar experiences.
You’re right about density, but I think there is a strong social/psychology aspect as well. As a hunter/gatherer you’re transient. You have a short time horizon and as the game moves, so do you. Liberals have this worldview. Go with the flow. Something for nothing. Anything goes.
An agricultural society is the opposite. You have to have an effective social construct based on shared, proven beliefs. Hence marriage and family are so critical. You also practice the Law of the Harvest - you reap what you sow. This induces long term thinking and the close relationship between risk and reward, hard work and effort.
There are exceptions to both of the above, but overall hunter-gatherers have high leisure, tolerance of the unconventional social norm and female oriented labor burdens. Just like the modern liberal worldview.