Is it really possible for a handful of Europeans to wipe out a culture? I doubt it. The people themselves were offered a better, more rational, and more humane worldview and embraced it willingly.
The major factor in the European conquest of the Incas were bacteria and viruses. Diseases killed a LOT more Indians than the Spanish, in fact the germs beat the Spanish to Peru, precipitating a civil war of succession that was in full swing when they arrived.
Also, the Inca, much like the Aztecs, were a relatively new and barbaric people who had conquered and were exploiting much older and more civilized peoples. They saw the arriving Europeans as an opportunity to throw off their overlords, with their small numbers meaning they were no real threat to replace them.
Big mistake on their part, as it turned out. But most of the actual overthrowing of the Aztecs and Inca was done by their rebelling subjects.