Logically, you are correct. The problem is, this is an emotional attachment to the Mount. I don’t know, but it seems to be the foundation is intact.
If reports are true, two things: 1). The mount was actually the Roman fortress that their garrison was housed and stationed at, 2). it was someone who visited old Jerusalem, way back before 1000 AD, who made the determination that the mount where that fortress stood was the site of the temple, and not any true Jewish historian or expert.
I'd have to break out my historical books, but even Josephus said the Temple was completely destroyed and no foundation stones remained.