If its in your divorce decree then he's right. Your decree should cover this.
My husband's says "Each parent agrees not to have an overnight guest of the opposite sex in the presence of the minor children."
Legal ways to get around this:
1. If you two aren't ready to get married, have him sign a lease agreement and call him your "roomate".
2. Get married.