Context matters in college admission. You're a good citizen who has achieved a good GPA and average test scores. If you did all that despite living in a home where the adult(s) aren't college-educated, and where money was always very tight, that's a more impressive achievement than it would be for someone who did the same thing with two well-educated parents and plenty of money.
A higher percentage of African-Americans are still growing up in tougher economic circumstances than most white Americans. Affirmative action is designed to help colleges ensure that that difference shrinks over the course of future generations. If you fit the norm, and most of your white friends have more money and better-educated parents than you, you don't have to apologize for taking advantage of an affirmative action admission decision or feel like it's unfair for you to have that advantage.
Of course you may have white friends who have even fewer economic advantages than you. In a perfectly just world, context would matter for them too. Here's hoping that's true.