The Seals were at the annex...the militants attacked the consultant first and came with diesel fuel and fired everything up without finding Stevens and Smith and an agent in the safe room.
some hours later they attacked the CIA annex and that is where the two Seals were killed.
The Special report is reasonably detailed in the attacks progressing thru the compound.