Adultery is a tort. The deceived spouse, or the children should be able to bring a case in civil court, seek damages from the other party, and should a divorce result, make the adulterer suffer in the financial settlement and in custody decisions.
Extramarital activity of unmarried adults is their own business.