Having lived in both worlds I would say it is the hard work.
If you live on a dairy farm you work half a day before you go to school in the morning and another half a day after school at night.
An after school job for a city kid will rarely compare in in any way to the kind of hard labor a dairy farm kid experiences.
Given your experience, what would you estimate the chances are of spending a day loading and dragging a manure spreader around a pasture and not ingesting some cow feces in the process?
In part that is true - Throwing hay bales will do far more for you than what you get at the gym... but I was pointing to the food - farm kids eat real food, straight from the ground. WAY more nutrition. That's why they have the energy to do all that work.
For the first time in some years, I will finally have a garden in this year, and I really can't wait. The food is SO much better than what I have to buy in the store. Already using farm bought eggs (and raw milk) - I hate them pale-yolked things that pass for eggs in the store - Once the garden is there, I will be raising my own chickens too (need the garden waste to feed the chickens).
I too lived in both worlds. The city made me sick. Literally stressed out, fat and ill. I am going back to what works, and the closer I get, the better I feel.