"Is," because as a pronoun "what" needs a referent, and here the referent is not the plural entity "people" who are responsible for my being bothered, because it has yet to be established that I am in that state. The referent for "what" is "the condition that you are about to learn has caused me to be bothered," and that condition is "the existence of people whose English is so poor."
I perhaps for completeness could've written "What bothers me is the subset of people whose English is so..." or "What bothers me is that there are people whose English is so..."