“The Fifth Amendment does not forbid adverse inferences against parties to civil actions when they refuse to testify in response to probative evidence offered against them.” Baxter v. Palmigiano, 425 U.S. 308, 318 (1976).
And once you get hit with adverse inferences, you can also lose attorney-client priviledge