If you don't feel comfortable don't go. You will be distracted the whole time worrying about the kid.
I ride with new riders a lot though, it's mostly wifes or girlfriends who took up riding their own bike.
I don't have a problem, we all just give them some room.....and they should ride in the back of the pack.
You are not a douchebag for being concerned.