I've heard that Jaden's/Judai's voice actor moved somewhere outside of New York, but I heard it during season three, before they even got to the Dark World arc. While I don't like it when voice actors are replaced, my opinions on the whole Pokemon situation alone proves that, but I don't mind it so much if either the voice actor/actress either doesn't want to do the role or heaven forbid passes away. If it is true that his voice actor quit, I don't see why 4Kids couldn't just get a replacement for the last season. Besides that, the majority of the rumors I've heard all come down to 4Kids wanting to dub 5D's and this is the first time I've heard of this.
Trust me, I've heard a lot of negative comments about season four, including the ending. I've actually heard some people consider it one of the worst endings to an anime. Regardless, I'm still willing to watch it for myself in English. I seriously doubt that it can be any worse than season one was.