Ok, that whole, do him and get it out of your system is just stupid, don't do it. i say you should try to ignore him a bit, and maybe just focus on the things you find attractive in your husband, and why you married him. If your attraction to his brother is purely physical, just make love with your husband more often than you do now. If it persists and its harming your marriage, than it's time to talk with your husband.
Hope I helped. Good luck.