Seems like you’re beginning to sour on this relationship. If so, I say good, I think he is using you and gaslighting you for sex. Sorry if that sounds harsh, I don’t trust him at all. Good luck with whatever you decide.