What do you do when your boyfriend finds out a bad secret about your past?

[BuTtErCuP]

Many,many years ago when I was a teen I slept with my boyfriend's brother. It only happen once and I did not know my man at the time. Well he found out I think his brother told. It isn't something you tell a guy when you first meet! I wanted him to give me a chance with out knowing that information. I met my boyfriend about 11yrs after I slept with his brother. My man and I have been getting super,super close. We have been hanging out with each other about every day now and telling each other how much we love the other. It has been 110% great! All the time when his brother is hanging with us he messing with us saying why are we together and we shouldn't be. He jokes with us tho and we laugh. I don't know what happen his brother had always said he wouldn't tell. I know his brother has been seeing the way me and my man have been getting serious. His brother is married and so I know it's nothing like him trying to get with me. Earlier my man sent me a sweet text! Then a little after that was when he texted saying what he heard. I know he's out drinking with his brothers right now. Anyways so now my man keeps texting me telling me off saying it changes everything. I don't want to loose him. I just kept texting back saying it's not true. I'm not sure if I should act mad like I'm the victim. Or just not text back if he textes me something stupid. Or what,one thing for sure I am not telling. What should I do? What should I say or act?????