Immigration is part of this but another major reason is that affluent families tend to have less than two children. So they are not even replacing themselves. However, it is not unusual for poorer families (or single mothers) to have several children raised on government money. These children grow up with a sense of entitlement as well as less education and less of a work ethic.
Ronald Reagan's prediction may come true, where there are more people riding in the wagon than pulling the wagon. We still have time to reverse this but there is not much room for error. Demographically, we are heading down a very bad path.
Blacks should be concerned about the 2050 projections as well; they will be living in the same Asian/Hispanic country as us.