If by "we" you mean white people, no, because we haven't been reproducing at a geometric rate. It's the black, brown, and yellow people who have been starving to death and suffering under murderous regimes. But the "no reason to limit birth rates" crowd never seems much concerned about that. As long as "we" aren't starving, the rapid increase in the world population, and its accompanying starvation and murderous regimes aren't a problem, right?
I care now, but everybody will care when it reaches a point where the regionalization of the problem that has held up so far, breaks down, and the swarms of hungry illiterate masses crossing borders into comfortable countries get too big to be managed at all. Then "we" will start starving and living under dictators like Mugabe and "Dear Leader". I'd rather turn the tide now. If we wait too long, we won't be in any position to try to improve the standard of living of "those black, brown, and yellow people"; we'll just be another color swatch in the global mass of misery. And the notion that we should try to compete with the other color swatches, by reproducing faster than they do, is beyond ludicrous. That will just result in us being a larger color swatch in the global mass of misery.
I think governments should just be allowed to kill off "excess" people, like the Chinese do. That will solve all the world's economic problems.