Apparently it is recommended that you keep cats inside for 3 - 4 weeks to make sure that they know where their home is. As you have had yours for four years, he does know where he lives! I can understand your worry, so you could let him out and go in the garden with him just before you are due...