The most convincing CGI I have seen in recent years has been from the clips I have seen of Where The Wild Things Are....only they weren't entirely CGI but men in costumes with CGI'd facial features.
The reason why this proved more successful than pure CGI was because the creatures possessed a real physicality and it is their physical interaction within a real world environment that makes it work more successfully.
Film makers invest so much time and money into developing hyper-realised CGI characters, and whilst these may seem more and more convincing with the development of technology, their shortcoming become evident when placed in, and forced to react within a real world environment.
It might not necessarily be the animated character that fails to illude us but it is the animated characters' reaction to a real, three dimensional environment is often unconvincing, it lacks physicality and it's interactions may seem subtly inconsistent, thus breaking the illusion. It is extremely difficult to merge the two dimensional and three dimensional effectively and pass it off as reality.
Perhaps the reason why Avatar has a more convincing CGI is because most of the movie appears to have been filmed in a two dimensional world and there is less inconsistency, however there is then the problem of trying to convince the viewer that they are in a true, three dimensional world when it is apparent they in a two dimensional animated world.
I guess to sum it up, CGI is more convincing when used sparingly, to embellish the three dimensional rather than to try and compete with it.