I believe that it's a bit of both personally, although I know some would argue.
I think about it like this:
Our genes determine if we are tall, fat, skinny, short, acne prone, deformed, etc. So in a way it determines if we are going to look attractive to others or not. And when you are not "attractive" to people it somehow affects how the react to you and how you feel about yourself knowing that you arn't the "hottest" person.
Based on that, if you already have low self-esteem because of that, a crappy childhood doesn't help at all. People are mean.
Put it this way:
When a beautiful person smiles at you, you go "what a nice person" and smile back.
But when an ugly person smiles at you, you wonder "what do they want?" and avert your gaze.
lol. Either way, your experiences define your personality, and your genes determine what possiblities there are for what you can do.