Because you do work on the swing by applying a force to the seat which is "in tune" with its oscillation like a pendulum.
You use conservation of momentum to do so. Your feet are the body against which you propel your torso and seat when you pump your legs.
The tension in the swing's ropes or...