What has to be retained is the concept of property, even as that property goes from being something kept in private to something availed on the open market.
My car is my car, whether I am on private property or public roads. The law requiring me to wear a seatbelt applies only when I'm on the public roads, and though I may not like it, I am compelled to comply by the nature of the property in question: the street is owned by the city, so they make the rules there.
But a restaurant isn't the same. One need not - our ought need not - necessarily parlay his entrepreneurship into the truly public sector in order to exchange goods and services among others. There (rightly) is no law stating that I must entertain black guests in my home. Why, then, should I be forced to entertain them should I open a restaurant? It's still a private building on a private lot, no different than a home.
It's a deadly thing to assume (or for that matter to conclude) that one's ability to operate on the open market be predicated on the willingness to sacrifice one's values - backward as they may be - for those of the masses. That constitutes a fundamental undermining of the "free" in "free market."
In other words, the answer to your question is "yes," particularly insofar as any such burden is undue, so to speak.