I can walk just as fast with my hands in my pocket or carrying a grocery sack. When I stop swinging my arms conscientiously, then forget to pay attention, I find those things swinging again.
It's a natural way of maintaining balance. When we walk while carrying something, or with our hands in our pockets we tend to have to concentrate more than usual about our balance. The fact that you stated "then I forget to pay attention" proves my point.
