It's just instinct I guess. I'm not sure how long your fish memory lasts but perhaps he recalls that experience. I don't think their memory is that great though, so I'd say it's instinctual.
Today, I was cleaning my fish's tank and i forgot I didn't have a net so I had to use my hands to get him out to clean. After all that, I put him back in the tank (with my hands) and now when I come close to the tank he goes the opposite way or hides. He's never done this before either. Is my fish scared of me?