Yes you can. The simple rule is: your rights stop at the other persons nose. In other words the minute the expression of your rights damages or interferes with the rights of another, your rights stop.
I agree with this rule. This is the litmus test I would apply. It does not necessarily have any basis in morality.
By the way, what you said does not contradict anything I said. You have further backing than saying "you just know."