My rights are violated when other people abuse our liberty. Therefore some of that liberty has to be taken away.
It's their's to abuse. How does it directly harm your rights? I can see crack use, or meth use as such an imposition of danger that it could be considered harmful to others, but pot? No worse than alcohol, and alcohol can be far more abused than pot.....