There's a big hill right next to the Great Zimbabwe ruins- it seems like they would've built the "observatory" on the hill instead of down lower. When I visited these ruins we were told the tower was possibly to contain grain and it was within the enclosure in order to protect it. Makes sense to me.
H Rider Haggard's "She" is inspired by these ruins too I believe (good book by the way).
