Risk of injury- especially if you don't wear a helmet or if you're sharing the street with cars.
Expense- a high quality racing bike can be very expensive- in the 1000's of dollars, though you can get a bike for much cheaper.
Weather- it can be tough to ride in bad weather, and sometimes...