Depends on the way you're raised most of the time, if you're parents teach you not to ride with people of any race other than your own, you assume it's the right thing to do.
Your particular area could suffer from significant racial segregation and people round where you live are fascists, discriminating etc.
Anything else is specific to the person's personal circumstances.