It's your wife, I can't see any problems with that. Unless you are being intimate with someone else other than your wife then it's a sin. Also, why do you have to tell your pastor about it? It's a sin for telling someone about your intimate relationship with your partner, even to your pastor. Only you, your wife and God should know about it.