Affirmative action hurts blacks too ---I have a couple of black friends who are extremely qualified in their jobs, have all the education and then some that the job required but had people assume they only got hired because of that. People saying things like "aren't you thankful for affirmative action" which really burned them because that wasn't how they got the job.
And in a more subtle way, it hurts them by making excuses for them, for enabling their continued failure, for patting them on the heads and saying "There, there. It's okay for you to make all the wrong choices. We'll get you a job anyway. We'll force your way into society at the point of a gun if necessary."
The presumption is that blacks are somehow unable to perform to the same standards as whites, and that this "handicap" can only be overcome with governmental assistance. It is the heart of racism to imply that blacks are so inferior that they can't succeed on their own. Yet that is precisely what "affirmative action" (which is anything but affirmative) does.