Every culture has their own rules about how people should interact. They are generally learned informally, and for the most part unconsciously, growing up in a particular culture. Some people learn them better than others. Some people's home environment was perhaps a little different so that they learned something a little different.
If people have mentioned it to you . . . well, the good news is that they care enough about you to mention something that they think is a little outside the norm . . . maybe you would want to have a talk with a life coach or a counsellor of some sort, maybe a social worker (not sure what's available in the UK) and see what someone "objective" thinks.
Kiasmama mentions some cultures not making eye contact. It is so true . . . like some cultures stand closer together when talking, some farther apart, relatively. It is interesting to study. In the US, some think that one (or the numerous) factors that perpetuated tensions between Caucasians and African Americans is that African Americans had learned to make less eye contact initially especially with people in authority (a legacy of slavery, I believe). On the flip side, Caucasians interpreted this behaviour as a sign of dishonesty or deceit.