It depends on what military branch your intrested in, here's there difficulty in order from hardest to easiest (physicaly)
Marines-Army-Navy-Coast Guard- Air force
and for the police branches they are also in the difficulty level
Csi/Fbi- Swat-K9 police-State Partol(Troopers)- Highway patrol- General Patrol
And just remember that even though they are all hard physicaly but also they are mentaly challenging.
