I don't know much about building up arm strength, but if you're using the lightest bat you can possibly use, you late swing might have more to do with you technique than your arm strength. Have someone, like your coach or an older player, help you with your swing. Make sure you're not dropping your hands when you start your swing, like a lot of young players do without even realizing it. You might just be starting your swing late, so make sure your hands are all the way back when the pitcher is releasing the ball and you plant your feet in time to make your swing.
Or if you take practice swings with two or three bats in the batter's circle before you step up to the plate, your one bat will feel A LOT lighter in your hands.