If you are a batting against a starting pitcher, your job is not only to hit, but to get on base. Swinging on the first pitch takes away a possible Ball one. First strike pitches are thrown just above 60% of the time, and because of that, swinging at a first pitch is iffy. You want better odds. You want the pitcher to earn his pitches..
If he throws a ball first pitch, you are forcing him to throw a pitch that you might expect on 1-0 (strike), 2-2 (off speed), etc. When you swing at a first pitch, you are also forcing your body to time the pitcher's first pitch. Instead, you are better off waiting for a good pitch (as long as it's not 1-2, or 0-2). When you swing at a first pitch fastball, your body has timed it by the time the play is over, and the next pitch might throw your body off.
Another reason is pitch count. As a baseball player, it's your job to get on base. The longer an at bat goes, the more chance the batter will have to get on base. And the more pitches a pitcher throws to one batter, the less he will be inclined to pitch to the next, forcing a bad pitch here and there. If you can run a starter's pitch count up, you can take him out of the game early even if you don't score alot. You want to earn your walks.