Indeed. However, where do we decide how far those right's extend? Sure, it's easy to say, "That which does not violate the rights of another.", but that is a paradoxical statement. To illustrate this, it would be similar to me answering the question "Who all is a female?", with the statement, "Whoever isn't a male." Well, it's not that simple. There are hermaphrodites, "gender benders", sex changes, ect. It is not cut and dry. There must be a precise definition if you want a precise answer.
I believe our founders knew this and gave us the 9th amendment.
The enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others retained by the people.
In this, it tells us "others retained by the people." Now. Where do we look to define what those "others" are? I believe the founders set up states for this purpose. You see, many things that the founders supported prohibiting in their own states, could be construed as rights by others (witchcraft, sodomy, ect). So, in order to enjoy those rights, not specifically identified in the BoR, one must live in a state with others that share the same view on whatever it is they see as a right; be it drugs, sodomy, prostitution, incest, and what have you.