It reminds me of the old joke where the patient says, "Doctor, it hurts when I do this," and the doctor says, "Then stop doing that."
It sounds like you're setting the tone by saying "something mean in a joking way". If you don't want him to respond that way, maybe you should stop doing that.