The irony is that he can say and use the term “my people”.
But if you or I uttered the term “you people”, that would be racist.
There's another problem with Holder's use of the term in this context. He is a first generation American, the son of immigrants from the West Indies. So he is being disingenuous when he claims the black victims of slavery, segregation, and denial of voting rights in the United States as "my people." His forebears never suffered from such racial oppression in this country! (The same holds for Obama.)