The people who are manipulating the system are the elites, not the minorities.
Think about it. The children of the elites (those earning income in the top one tenth of one percent) are well connected and well financed. However, they are not necessarily intellectually brilliant. Above average, yes. Brilliant, no.
Competing against them are the children of the vast middle class, some of whom ARE quite brilliant. So, how to cripple the future of these children, to prevent the emergence from among them of another Steve Jobs, Bill Gates, etc. How to reduce the chance that somebody brilliant will come along and displace them?
One way is to saddle middle-class grads with so much student debt that they will be forced to immediately get a regular job, rather than have the freedom to contemplate a startup which will bring one of their bright ideas to market?
The non-brilliant affirmative-action student poses no threat to the elites. The brilliant-but-middle-class student DOES present a threat.