The fans don't make the same amount of noise when Russell Wilson is calling out the cadence as they do when the opposing QB is.
But the defense is on the field when the crowd is making noise against the opposing offense, so their communication is hampered. And the offense is on the sideline talking about the last possession. Noise effects everybody.