How do they test mental unstableness?

[DeViL_sPaWn]

Say if a teen kill a few people on purpose, how would the courts and stuff decide if they are mental and so can't be responsible for their actions?

And what would happen to the teen if they were found to be mental?