It's more the impact than numbers.
Remember elementary school? In a class of 20, the 2 troublemakers were sufficiently diluted. If the number of troublemakers were 4 or 5, the entire class would be disrupted.
Even if the number of "moderate" Islam is 70% or even 80%, the impact and numers of the remaining "troublemakers" is catastrophic.