Why don’t you just keep things as they are? He’s not going to leave his family, neither of you are satisfied with your spouses, so maybe in this case continuing this is the best way forward. Your spouses must have a sense something is going on outside the marriage since there seems to be no love or sex in either of the marriages.