Observation and inference, I should think, would only be necessary to move men to organize the growing of things.
Some wild fruit or something such as a pumpkin producing growth as a result of rotting away in one spot would be all it would take to fire the mind of any reasonably intelligent person. I’d bet that the degree of order imposed upon farming (and the resulting yields) might vary more than the fact of farming itself coming along.
Then why didn't men of 20 or 100 or 200 thousand years ago figure it out?
A legit answer is "they did, we just haven't found the evidence yet." It would account for how farming apparently developed at the same time all around the world. Naturally then the question is, why does it appear that farming developed so recently?