It's because all the "middle men" rob the farmers of any decent profit they can make. Til they've invested in all their equipment, land, etc., they just simply can't make enough to make a decent profit, especially considering it's a 24/7 job in most cases.
Easier and better money can be made elsewhere, so unless you REALLY have a love of farming, then why not do something else?