There are always limits to rights. The famous "fire in a theater" limit to free speech, for example.
As to privacy, I don't have a right of privacy to kill you, if I do it in my home. And as I posted above, I don't have a right of privacy to put substances such as cocaine in my body (although, I think I should have that right, it hasn't been recognized by the courts).
So what are the limits of privacy? In my opinion, I should have the right of privacy to put drugs in my body, well before someone has the right of privacy to terminate a late term baby, because in the drug case there is no question of another human beings rights (the childs rights) on the table. So I think the courts have things a bit backward.
Due cause and the commission of a felony are justifiable reasons. IMO you are correct in your take that there are limits. The courts seem to agree.
I do believe we do have a justifiable right to privacy as a rule.