This is a longstanding debate. In short, we're not real sure. St. Augustine took one of the more harsh positions, and he basically said that they go to hell but there undergo a punishment so mild, that it couldn't really be punishment at all. A couple of folks in the Middle Ages went even farther than that, but they were called "baby torturers".
Generally, however, the position has been that the unbaptized babies suffer no pain at all. Here's what Gregory of Nazianzus said:
It will happen, I believe . . . that those last mentioned [infants dying without baptism] will neither be admitted by the just judge to the glory of Heaven nor condemned to suffer punishment, since, though unsealed [by baptism], they are not wicked. . . . For from the fact that one does not merit punishment it does not follow that one is worthy of being honored, any more than it follows that one who is not worthy of a certain honor deserves on that account to be punished. [Orat., xl, 23]In recent years, the Church has also been a little more speculative on whether such babies might even enter heaven. Personally, I'm a little wary of that position as it cuts too close to condemned doctrines. But like I said before, it is certainly possible, particularly since we know that in certain cases the desire for the sacrament can be accepted by God in lieu of its actual reception.
We had a miscarriage last year. We would have baptized our baby if we could have, and I'm not entirely certain that that baby didn't go to heaven. I guess we shall see.
Babies are innocent. They have no knowledge of sin or right and wrong. Of course your baby is in heaven. Would our God send an infant to hell or limbo because they’re not baptized? Of course not, Claud.
"I do not doubt that the infants whom the Lord gathers together from this life are regenerated by a secret operation of the Holy Ghost...to say that the countless mortals taken from life while yet infants are precipitated from their mothers' arms into eternal death is a blasphemy to be universally detested." -- John Calvin
And later Benjamin B. Warfield wrote concerning those who die in infancy...
"Their destiny is determined irrespective of their choice, by an unconditional decree of God, suspended for its execution on no act of their own; and their salvation is wrought by an unconditional application of the grace of Christ to their souls, through the immediate and irresistible operation of the Holy Spirit prior to and apart from any action of their own proper wills... And if death in infancy does depend on God's providence, it is assuredly God in His providence who selects this vast multitude to be made participants of His unconditional salvation.... This is but to say that they are unconditionally predestinated to salvation from the foundation of the world."
God gives and God takes away. But we will all be together again one day.