Conservatism (at least today’s version of it) demands that race doesn’t matter, doesn’t exist and/or must be ignored.
That goes against human nature. By erasing the Hispanic/Black/Asian/etc... groups you erase a large part of people’s identity without replacing it with anything of equal value. Race/ethnicity identity runs deeper than political philosophy.
That sense of community that was taken from White society hasn’t been taken from any other race/ethnic groups and they reflexively refuse any attempts to do so, even when it may hurt them in the long run.