If you didn't know this before committing to marriage you were very naive to the situation. I think at this point you make the best of it or leave him after telling him exactly what you've told us. The fact of the matter is simple...if the gym is more important to him than time with his wife...
He's a creep, he disrespects you and he wants laid by a different gal. If you didn't realize this before you really need a wake up call. He sounds like trash actually.
With children as young as yours to be able to have it as often as you do is incredible. There's not much you can do since most married people in the same predicament will tell you they don't get it near as often as you currently do. If you are wise you will listen to your wife and back off...