Slavery had already been abolished by the 13th Amendment before the 14th was ratified.
But that doesn’t mean they meant to include foreigners. Perhaps they did, but both in the original and in the 14th, it’s not clear-cut.
The wave of European immigration to the United States began in the 1830s and accelerated after the Civil War; by the time the 14th Amendment was ratified in 1868, there were already enough Irish immigrants in Boston and New York City to substantially affect the Congressional representation of those cities. If Congress didn't want aliens to be counted, they would have said so.
The debates used "inhabitants" and terms like "whites"...and with the Great Compromise and finalizing the document, it went to "Free Persons" because that would cover free blacks, etc.
And the 14th Amendment was a modification to the original language. By using the same language, it wasn't some signal that now we were going to start counting non-inhabitants! Or foreigners!
Just think of how absurd it would be to use your interpretation. If the Census had fallen during the War of 1812, I guess we'd have had to give representation to British troops on American soil. Does harboring a fugitive from the law mean that your family should get greater benefits from the government? Etc.
Remember also, Tammany Hall worked hard to get citizenship for those immigrants, wanting to gain power from that demographic. Combine that with the natural-born citizen clause, the foreign title clause, etc., and you realize that there was never any intent to give foreigners power or representation in our governance.