Under most circumstances, I would agree with you. I myself, am a high school teacher and I have seen many peers vilified for out of school conduct which, while not illegal, was considered immoral and which I thought to be innocous. Like going to a "swingers Club" with a wife, or other things of that nature.
BUT this NAMBLA thing is different. Just because he hasn't ever been accused of official misconduct, doesn't mean he hasn't done it. NAMBLA is a predatory organization composed of predatory pedophiles. They actively support and act as conduits for this atrocious behavior by suggesting means of remaining covert while continuing to prey upon the youngest and most helpless of our citizens. That such a person is not only a member of this group, but who obviously feels secure enough in his membership to publicly support their goals.....all the while placing himself in immediate proximity to our children is enough to send chills down my spine!
SCOTUS is made up of humans.
Humans are subject to poor judgment.