I meet all sorts of women while they are with other men (married or dating). Every man does. If I later start a relationship with one of them, after she has parted ways with her prior lover, how does that make me a bad guy?
All we know is he MET her when she was married to someone else, not that he began an affair with her then.