Think of the striking differences in each groups views of property. Plus, the productive density of agriculture would give additional calories to farmers who could always supplement by trapping and hunting.
Too much leisure is a net cultural/societal negative.
Big issue is density. Agriculture can support, in most locations, a population 10x, 100x or more that of hunting/gathering. Look at the populations of major American Indian tribes in 1700 and the absolutely immense territories they, sort of, controlled.
In the long run, this means the farmers will win.
Just finished Guns, Germs and Steel. An interesting book, though Mr. Diamond had an obvious “anti-white” axe to grind.
When he spoke of “white” expansion into Australia and the Americas, he use words with negative and judgmental connotations like “invasion” and “conquest.”
When he spoke of similar expansion by non-white groups, such as the Bantus into most of sub-Saharan Africa and the Austronesians into Indonesia, he use neutral, objective language such as “expansion” and “spread.”
It is likely the groups displaced had similar experiences.