This is considered a historical fact.
Historical facts count as knowledge which can be as dependable, and in some cases more dependable, than facts arrived at through scientific experiments.
That the earth is billions of years old counts as knowledge in my book. If you want to argue about whether it is scientific or not, great.
People who call themselves scientists (but maybe in your book should call themselves historians) have made careful measurements of geological formations along the west coast of Africa and the east coast of South America. They have discovered similarities in the formations such that they can match rocks near the west coast of Africa with rocks near the east coast of South America. The evidence suggests, with a very high degree of probability approaching 100%, that Africa and South America used to part of the same continent.
These same "scientists" have placed instruments on the two coasts and make regular measurements of how far they have moved apart over the past few decades or so. Using that information they have extrapolated back to get an estimate of the time when the two continents were one. Their estimates suggest that millions of years have passed since these continents were one.
Everything we have learned about cosmology comes from data that was generated in the past. We can't conduct many experiments in cosmology, but scientists can make guesses about what they might see in the future based on what they've seen in the past. If those guesses turn out right time and time again, then those guesses mature into hypotheses and then into theories. This is what seems to be happening with regard to more and more careful measurements of the cosmic background radiation. Recent developments suggest that there might now be sufficient evidence to back up the Inflationary Model of the early universe.
This is all good news to my mind. There is some knowledge that is clearly "scientific" in the way you mean it. There is other knowledge which is clearly "historic". There is other knowledge which might be some combination of both. Why would we want to dismiss this knowledge because it isn't 100% scientific? I watched a recent documentary attempting to determine "Patient Zero" for the Spanish Influenza outbreak that killed so many people around the time of WWI. Historic documents and scientific evidence were brought to bear on the question and the conclusions that the "scientists/historians" came to seemed quite convincing. This knowledge might help us more quickly identify the future sources of outbreaks or respond more quickly to prevent them from becoming epidemics. What's wrong with that?
Going back to the theory (or hypothesis?) of Continental Drift, yes it is possible, but highly unlikely, that the two continents separated in a few weeks or months or years, but a vast quantity of carefully collected and analyzed data suggests it took millions of years. Call this knowledge historic or scientific, it still counts as knowledge.
I agree with you that some scientists are full of hubris. There's that egotist Krauss who wrote "A Universe from Nothing" where he purports to have shown exactly that. However, every respectable philosopher that has looked at the book has found it laughingly bad. There are the neuroscientists who are certain that we have no free will and no "self" and that we are just AI machines made from biological material that will someday be replaced by better AI machines made with more durable stuff.
Besides these few showoffs, who seem to love being in front of cameras more than being behind microscopes or telescopes, there are numerous scientists who spend countless mind-numbingly boring and backbreaking hours collecting specimens, categorizing them, analyzing them, and coming up with hypotheses as to why they are one way rather than any other. Some of the knowledge they provide us can be clearly identified as "scientific", some as historic, and some as a combination of both. It is still knowledge and still a reflection of the ultimate source of all knowledge: God.