Avoiding taxes is going to be the big stickler. If you grow some in your garden and it makes it onto the street, how is the government going to know? If you grow 3 lbs of pot and it is selling for $150 an oz, are you going to turn that down?
We regulate other mind-altering substances when they are brought to bear in the marketplace. While one may brew beer in his own home, and even sell or give it to his neighbor, there are legal consequences for public sale without having proper authority. I happen to believe alcohol is overly regulated these days, but it needs regulation.
We ought to realize cannabis only a plant. Good heavens. If I had the tools to grow it (which really are not complicated at all - seeds, water, and light), I might as well give it away. We’ve attached too much importance/value to a plant and its effects. Let people be judged by their actions, and establish limits when it comes to partaking of substances that induce aberrant behavior. I tend not so much to blame the substance, but the heart and mind inclined to worship false gods, which are far more abundant than cannabis.