I agree that the accents are very difficult to understand in some places, and yeah, they completely failed to make Matt Damon look tall (I didn't realise until I read about it afterwarRAB that they were even trying to do so).
I wouldn't say the film was rubbish, though, far from it. I really enjoyed 'Invictus', it was a classic piece of Clint Eastwood film-making: get a solid script, pick some good actors, step back and let them get on with it.