Is there pain or discomfort? If so, then you really should see a doctor! Scrotum shrinking happens when stimulated sometimes and when it feels the temperature it is exposed to is too cold. Shrinkage happens when your testicles need to be close to the body for proper warmth.