Growth won't be the issue, aggression will. When they grow these cichlids get much more aggressive, and in smaller tanks even more so. If you havea 55 gallon tank, there is no reason not to move them now.
It is also better to move them now because you can add more africans while they are still young so they can grow up together and get along better. Also because a 20 is cramped and inhibits activity - these fish are very, very active.
You do understand, I hope, that "african cichlids" covers a lot of different fish, and not all - in fact most, are not compatible. Choose species wisely, research them, and remember that what works when they are juveniles will not necessarily work when they grow. there are better and worse ways to go about stocking these guys - randomly choosing various species without looking into it is the worst way.