Appeal - The broadest and most important factor of them all. At it's core, animation is about bringing fantastical ideas and concepts to visual life, whether it's presented as a consistent narrative or an abstract piece. Appeal is nothing more and nothing less than asking: does it effectively express those ideas and concepts? What appeals to each individual is ultimately within the realm of subjectivity, but as far as I'm concerned, a vast majority of ideas and concepts can be expressed properly and effectively even if I don't usually care for them. Appeal ties into every aspect of animation production, from art direction and character design to characterizations and plot ideas to the soundtrack and voice acting.
Fluidity - The technical quality of animation (the italicized distinction will be crucial for the next factor). How many frames are there from pose to pose? How articulate are the poses themselves? Not everything can have the animation budget that the WB Silver Age shows did, so IMO a lot of contextual leeway is necessary here (rather than acting all like John K. and bashing everything that isn't full Clampett-esque animation). It's ultimately up to whether or not the animation is appropriate for whatever concepts or themes the show is expressing. South Park doesn't need full animation. Home Movies doesn't need full animation. Dan Vs. doesn't need full animation. These and other similar shows aim for a limited aesthetic in favor of emphasizing dialogue and/or character-driven material. And finally, the level of design detail =/= fluidity of animation. This is important because it seems like everyone confuses the design aesthetic with the actual animation process. Adventure Time has simplistic designs yet some of the smoothest and fullest animation in today's cartoons.
Acting - Obviously, this has to do with the quality of voice acting -- however, it also has to do with the artistic quality of animation. As far as I'm concerned, the character designers, storyboard artists and animators are acting just like live-action actors would if it were a live-action production (that's a tongue-twister!), as they're taking all the concepts and ideas from pre-planning and/or a screenplay and molding them into it's final visual form. And from there, voice actors and actresses are the final piece of the puzzle; do they deliver a quality performance? Are they appropriate?
Wit and/or Pathos - The effectiveness of the comedic and/or dramatic elements of the show. In other words, this is one of the outcomes of the acting employed to express the appeal of the ideas. Is a comedy toon genuinely funny? Does it approach humor in a clever or unique way? Do the characters exude an infectuous charm? For more dramatic toons (almost always action), am I engaged by the dramatic events? Am I compelled by the struggles of the characters? In both comedic and dramatic toons, do I care about the characters? Finally, wit can apply to action/drama shows (mainly through comic relief) and pathos can apply to comedies (IMO, the characters should have a bit of depth beyond their designated shtick).
Atmosphere - This is another key result of the acting employed to express the appeal. Am I immersed in the show's fictional universe? If it's a comedy toon, does the atmosphere add to the humor and charm? If it's an action toon, am I engaged in the mythology that's being created? This is where the vast majority of Flash animation (not made by McCracken or Faust) fails miserably; there is NO atmosphere at all due to a near-total lack of appeal.
Continuity - Believe it or not, "continuity" is not limited to "story arc" or "serialized storytelling", so this applies to both comedy and action shows. This is fairly self-explanatory for action toons; are the plot points consistent? Do characters develop in a believable manner? Is the resolution to a long-running storyline satisfying? For comedy cartoons, this has to do with whether or not the show obeys the established rules of it's fictional setting and if the characters behave consistently while still displaying occasional depth or nuance. I also like it when they do subtle yet clever throwbacks to past events, something Foster's was very good at in it's later seasons.
Sincerity - This is whether or not the creative minds behind the show have any honest creative passion for what they're doing. Are they hungry for crafting a rich universe, a dynamic art style, fun characters, witty gags, compelling stories? Or are they simply coasting on their animation expertise for a steady paycheck? This is why I love The Secret Saturdays and dislike every incarnation of Ben 10. Hell, this is why I respect something like Jimmy Two-Shoes much more than Johnny Test or similar Canadian flash toons. I can easily tell what's a labor of love (even if it's flawed or bad) and what's nothing more than soulless hackery.
On the flip side, the year's young and something new could come along.