Agree to hang out with them, then instantly call them back and cancel any plans and your rating won't go down at all. Don't think this was common knowledge, but I don't do shit with anyone after I get their special ability, but maintain my friendships by canceling on the plans when they call me.