Cats love the outdoors its instinct and if he is used to going then he will persist until he gets his own way,
but neighbours can be funny about it but what we do is put a litter tray for our cats in the garden and they use that,i see them come home then go for a stroll again,
maybe a higher fence would do but knowing a cat they would find a way to get in to a garden,and cats are free to roam its the law that no one can harm them no matter what(it is here in UK)so if he loves it don't deprive him,
and if the neighbours are aware tell them to put a cat repellent down and even offer to buy some but if they don't know its your cat doing it then they cant blame it on yours without proof,
but try the litter tray in the garden its always worked for us.