Not really, he probably could have been distracted, showing respect, or maybe he doesn't like making eye contact. Get to know him better. If he doesn't sound as if he knows his stuff, that could be a bad sign.
When i shake hands with a person that i just met or don't know well, i look away too. it's a unirversal jester that means i will give you the benifit of the doubt and will share my peace with you but i will not give you my trust until i know you better. if the person is well known to me i will give them my full attention and respect.
If he was like me, don't take it personally because it is the way of a true warrior and when a warrior looks into the the eyes of a stranger it is because he is giving a jester that means; go for it.