If you can picture what is now called the Temple Mount to the north, the actual Temple stood to the south of it, down an incline of sorts. What people call the Temple Mount now was actually a Roman fortification where their troops were housed.
Check these out
Video 1
Video 2