Wait until you're 12 weeks. You're chances of a miscarriage have dropped a lot by then, and it will have given your brother and his wife some time to settle in with their child too.
Really, if you want to, tell everyone now! They're still going to be excited for you. I actually found out I was...