Probably. I don't think the jails in Australia are anything like the ones they show on American movies. A baby isn't going to know what is going on around him, he will just think they are visiting a new place and meeting their uncle.
I would take my baby if their father was in jail.