if you're talking british army then they will look at your medical history but assuming the reason you were in a mental institution has been resolved or does not affect you anymore then you shouldn't be turned away from the army, not sure about the US rules but I imagine it is pretty similar.
