It was after Prisoner of Azkaban that they ALL considered leaving. I think this is partly to do with what was going on behind the scenes, they were going to potentially take a much longer break between Prisoner and Goblet of Fire, do that as 2 films split up AND completely recast with more experienced actors. IMO whoever pushed to stop all that happening really had their head screwed on.
Emma was probably worried that she wouldn't be able to go to university because of Harry Potter, but things have been scheduled so she can go. I'm guessing if they couldn't have sorted that she may have left the role of Hermione. How mad would that have been though? A new actress in the last film?