Interesting that in Ark Gray rejected Elk when he himself wrote the majority opinion in Elk.
American Indian sovereignty accounts for the difference.
As Abraham Lincoln had once told a Judge who had bemusedly asked him why he had successfully argued one side of a legal point that morning, but was arguing the opposite side of that same legal point in the afternoon;
"Your Honor, this morning I believed myself to be correct, but this afternoon, I have realized that my argument is even more correct."
It's all about Politics. Gray was a Republican and a Northerner. (Boston Massachusetts) Most of the people who sided with him were Republicans and Northerners. The Two opposed were one Northern Democrat and one Southern Democrat. This was during the Era when the Republicans were still trying to stomp out Democrat racism, and I suspect the ruling had more to do with that than any real point of law.
One does not need to research all the way back to Calvin's Case when one has an Amendment passed just 30 years earlier. Gray was "rationalizing" a decision he knew would produce anger and controversy, and consistency was not something he was terribly concerned with.