People with the financial means, both white and black, move to an area with good schools because they love their children. If the inner city schools were the best they would not move.
That's exactly what Holder means by "subtle racism": depriving the community of your dollars just because your kids aren't tough enough to take an occasional friendly beatdown from troubled young persons of color. :)