Just feed him one type of food and don't change the water so often. Your fish can't like the temperature being different all the time. Also, he's probably a bit stressed, cus' you obviously can't change the water with the fish in. At least, it's not a good idea. If you keep taking him out, then...