It does and would make a difference. On a lot of courses you wouldn't be allowed onto the greens. There are actually "dress codes" on a lot of courses.
Actually a lot of golf is the "dress". I love the clothes...it makes the game what it is. My Hubby is African American and plays several times a week..he's all up into the clothes.
Peace, Love & Happiness