It can be done provided you have a lot of energy and you've planned it very carefully.The most important item you need is one tent or trailer that can sleep all of you together.The children need to be and feel safe in the darkness and sounds of night.All of the children need to be within a very close radius of their guardians. The younger they are, the closer they should be.Babies have been snatched from campgrounds by bears when they were only a few feet away from their guardians.This happened in daylight.
You need to bring a huge amount of provisions which can make setting up camp an arduous task and one which can take your attention away from the children. They have to be your first priority.
Campfires are another big concern. The fire also needs constant vigilance, therefore, I would for-ego the fire until the children are older and more responsible. A propane cook stove on a safely elevated table is a more appropriate option with toddlers.
Ideally, there should be an equal number of adults to the children in this age group and each should be assigned to a specific child so the attention isn't divided. I must admit that when my children were young, I would never take them to a remote or crowded place by myself for fear of them taking off in opposite directions.
With all that said,camping is a great experience for a family. I just hope that you don't get turned off to it because of the work involved .