Not if you are both mature adults. If the fragile ego protection rules of the child's playground are still in force (as they are in most of American society) then, yes.
I see all of us as merely disobedient children on the playground.
I am 54 and have seen SEVERAL of these relationships end up with rekindled affairs. And in every case the parties involved were otherwise very “mature” and responsible people.
I was once told by someone that once you’ve done crack there is no going back - the temptation will be there to wrestle with the rest of your life. Likewise, once you have had sex with a particular individual, you will ALWAYS see them as someone you had sex with. The ice has been broken. This does NOT mean it WILL end up in bed again, but the risk is substantially greater that it will than if you had never had sex. Obviously the risk is further increased if you are friends.
The caveot of course is if one or both is no longer sexually desirable which, as we get older, is more and more possible. But even then, the issue is not that you are more “mature”. The temptation simply is not there.
We could never have the relationship with my wifes friend and her husband had either of them been an ex-lover with either of us. Not possible, nor prudent.
Keep in mind, this all has to do only with exes with which you had sex. Again, it is like crack.