It's only with its entombment in Shia Law and its expansion beyond people with some history of civilized life that it got out of hand, and there we are today.
The evidence for this is the use made of Christian churches throughout the Middle East before 1000 AD. There's no sign of violent takeover like there is of Hindu temples throughout India and Pakistan.
It's actually quite a point.
There are forces afoot throughout Islam seeking the origins and trying to figure out what that means ~ one such item involved the Kaaba itself. The King of Saudi Arabia decided they ought to do formal professional archaeological work to investigate that building and try to identify who built it ~ at the time all they knew is that it had been there long before Mohammad's time.
National Geographic covered the work, and it's available on the internet. Not unsurprisingly it was a Hindu temple site ~ with a chief god that seems to have been a snake of some sort. There were a variety of odds and ends here and there that suggested very long use for that purpose.
I think the bit about the Ka’aba being a hindu temple is more hindu propaganda. There could be a grain of truth in it though...