Well, you have two choices.
To believe him, or think he's lying.
But as your boyfriend, I think he expects you to believe him.
Do you have any reason to believe he's a liar?
Otherwise, you have to give him the benefit of the doubt, and believe that they are his brothers, or whatever. How long...