They all act like they know everything and say 'all you have to do is say go and the horse does all the work and all you have to do is sit there and bla bla bla..'. What's the deal?
You just answered your own question. The reason people don't consider it a sport is because it isn't. All they have to do is say go and the horse does all the work and all you have to do is sit there. Pretty simple.