I think Id most hate to loose my sense of touch. I mean all of the senses are important and come almost equal in my eyes but if I was forced to pick one that Id hate to loose the most then yeah it would be touch.
However that answer is likely to change, if I were to answer tomorrow and it would probably be different.