That’s imply not true. A “drive by” attack can occur.
Any program (plug-in) sitting on top of the browser only has what access the browser allows it. If the plug-in exposes a vulnerability, then it is the browser’s fault. At least, this is the logic used to blame IE for add-ons installed there that create problems.