If that were the case, we wouldn't be having the debate.
Who other than the user is victimized by drug use - as opposed to rights violations motivated by the War on Drugs, such as theft done to pay drug-war-inflated drug prices?
To say that drug use affects no one but the actual user, however, is foolish. It's like saying that abortion is a "personal choice."
The rest of us are forced to choose the least odious of the alternatives.