Did a number of the Whites who would have had kids move somewhere else?
It is like Europe. Most whites have 2 children or less. The average comes out to barely above replacement levels. As older generations die off. The decline will really kick in.