Left to their own devices, they face predators, automobiles, fatal, highly contagious diseases and cruel people. They act as a reservoir for rabies, which is fatal to all mammals, including us. They do have a negative effect on the small wildlife in their area. They need human intervention.
I don't want to see packs of dogs running and killing deer. Game wardens are required to shoot strays that harass wildlife. At some point, laws may be passed to require the wardens to eliminate feral cats as well.
Sounds like they need to require people to spay/nueter cats in your county too. It all starts with people being responsible.