You miss the point. The so-called Moslems who just want to be left in peace are not really Moslems because they don’t take their Koranic imperatives seriously. They’re secular, for all intents and purposes. They’re not “moderate.” And they’re not nearly enough of them. And perhaps most importantly, they rarely publicly speak out against jihadis.
I agree with you because I know a number of Muslims who are actually secular with a generalized morality that would fit in several religions. The problem to society comes with their children who get more exposure to Islam and who identify as Muslam. They have the potential to become problematic.