You’re thinking of the Kaaba stone. That’s in Mecca.
In Jerusalem, the Dome of the Rock mosque sits over a rock from which - supposedly - Muhammad ascended into heaven. Here’s the rock: http://en.wikipedia.org/wiki/File:The_rock_of_the_Dome_of_the_Rock_Corrected.JPG.jpg