But you fail to remember the fundamental problem. It is not the government's job to represent the wishes of the voters. It is to represent the law and protect their rights. The representatives are there to represent the rights of those people. Not their wishes and not their designs on everyone elses freedom.
At least that's the way its supposed to be in a republic. I don't know what you have in mind but so far it isn't remotely libertarian nor is it constitutional.
This republic is endowed with powers that are delegated from me and you.
If I haven't got the right to take away your rights, then it is impossible for me to delegate that very authority to somebody I elect. The collective does not become stronger in that regard. Thus, every so-called action by the government which supposedly acts on your authority when in fact you didn't have that authority and couldn't possibly have delegated to anyone, is illegitimate.
You cannot give away what you don't already posess. Thus, the government which accepts power which nobody could give it, is an utter fraud. No. The government does not represent a population which "votes away its rights."
It merely pretends that the power it was "given" is real. All such power excercised is fraud and force.