I hear you. When asked in Spanish if I speak Spanish I invariably respond I only speak English, but I say it in French :)
From now on, I will respond: "Nyet! Ya tol'ko govoryu po-angliski!" (Russian for "I only speak English")