To believe man was responsible you would have to believe that small groups of hunter-gatherers in N. America were responsible for the destruction of mammoths, mastodons, giant bison, horses, camels, harrington's mt. goat, american cheetahs, american lions, american mt. deer, short-faced bears, smilodons, the giant cave bear, the woodland muskox, the giant ground sloths, dire wolves, the Stag-moose, the giant armadillo, and on and on and on...
The extinction list is way too long and diverse (including birds and reptiles) to simply tack it onto the butcher's bill of humans. Correlation is not causation.
Extend the argument to it logical absurd conclusion. If man could wipe out mega-fauna as soon as they arrived, then why are there any animals in Africa where man was present many many years before in the Americas.
If by small numbers you mean 100s of thousands to millions of the people than yeah they could have easily drove those animals extinct.
About 3 million years ago, Panama arose out the sea and connected previously separated North & South America. This allowed the Saber tooth tiger to enter South America where they immediately reeked havoc and drove pretty all of South America's Ungulates, marsupials and large predators like Terror Birds extinct.
Now if small numbers of saber tooth tigers acting individually can drive a large number of animals extinct then it's not much of a stretch to see groups of humans who compared to Saber tooths are much more efficient and diverse hunters doing the same.
Especially considering one of early man's hunting strategies was to cause stampedes and drive whole herds of animals right off the sides of cliffs.