Yes, you are looking at it wrong.
Any control on money is control of our right to organize, associate, and fight for what we believe in.
There should me no limit on who can contribute, other than it must not come from a foreign source.
There should not be any limit on the amount that can be contributed.
Money IS Speech.
Does that suggest that corporations should not be able to contribute?