Swinging is something that can be great for some couples, but disaster for other couples.
Yes, it is selfish of you to expect him to be ok with you having sex with other people, but you not allow him to have sex with other people. You have desires, you are curious, and so is he. There is no difference between you having sex with these girls and him having sex with these girls. NONE. Yet for some reason you seem to think there is a difference.
I disagree that its not real... If you are there you are really having sex with other people. But he right to a point that its just fantasy. I wouldnt use that word though. What he means by that is that he doesnt have feelings for these people, that its just the sexual factor of having sex.
And this is what most people who go into swinging miss.. In order to remain succesful as a couple while swinging, you truelly have to view sex as a fun activity that is in no way connected to love and emotions. Clearly, that is not how you view sex. If it was, you wouldnt have gotten jealous when he wanted to do the same things you were doing. Clearly, on some level, you still view sex and love as synonymous. They arent.
If you want to continue swinging, that is fully ya'lls own choice. But I think now you have a more realistic view of what emotions are going to crop up with it. If you guys cant keep your jealousy in check, it will destroy your relationship.
If you do continue, the two of you need to sit down and TALK and talk and talk and talk some more about it. You both need to be on the same page about doing it, WHY you are doing it, and what you expect to get out of it. You should also discuss some ground rules (like how to handle the situation you are asking about, as well as other ground rules).