If Rand Paul said "Yes, businesses should not serve blacks." That would be racist. Saying a private business owner in their private business should have the right to refuse service based on whatever criteria they want is far from making him a racist. You're looking at it too narrowly. It's not about a "right to discriminate", it's about a right to sell your products or services based on whatever criteria you want, because they belong to you. Be it you don't like people who wear hats or you don't like whatever race.