My boyfriend wants to start hanging out and talking to his ex.?

[Head]

My boyfriend and I have been on and off for the past 4 years and we have 2 kids together. We have been back together for about 6 months now. He has an ex-girlfriend (if you wanna call it that- more like ex- hook up friend) who he was with for about 3 or 4 months while we were on a break. I just found out that she has been calling him again wanting to know why he stopped talking to her and wants to start talking and hanging out again. We got in a huge fight about it because I can't stand the girl and I don't want him having anything to do with her. He said he has been friends with her and her family for 15 years and refuses to throw that away because I can't get over that they were together for a very short time. I even offered to go out with them but he said she doesn't like me either and won't go with me. I think that's messed up. Before they started "dating" he never hung out with her since I've met him. He said its because she was married and had her own life. I tried explaining to him that he now has his own family and she should respect that but he doesn't seem to get it. I asked him not to hang out with her but I don't know if he'll listen. Should I be worried?