True, but you weren't destracted by him in the first movie though. He might have been sort of mentioned, but not much. Which makes it easier not to think or worry about him. Darth Vader was the main villain and you felt his presence and he urgency of defeating him or evading him.
However in this movie, he was at the forefront of the movie - right at the beginning, making you think that there might be some action or threat coming from him, which never did materialize. So they kind of dropped the ball on that. I could understand it if it was going to go through the whole book series, but nothing was mentioned so if you take it as a independent movie in itself, it's hard to reconcile even seeing the guy. It probably would have been better to see a shadow than a real person. Plus, they didn't pick the greatest guy for the 'big villain' - talking about the king. He just doesn't seem threatening.