Your not really jumping or bouncing around with yoga. Wear loose, comfortable clothes. I wear sweat shorts and a halter, but my breasts are small too. I see no issue with a sports bra.
Hope you like yoga. It's not for everyone. (i hate it, but still go because it helps with Gymnastics). Only once have I been in a class with a man in it. Not common.