It's like saying we should let everybody out of prison because a small percentage are there due to a miscarriage of justice.
I say end the foodstamp program now. I'm sure the legit cases will be provided for through private charities. If my taxes were halved tomorrow, I'd definitely throw a few bucks to my local church soup kitchen and so would many others.
You allude to an excellent premise.
Big govt (and associated high taxes) is corrosive to charity.