In the 65 minutes that a sequence of power line failures built up to a cascading blackout across the Midwest, the Northeast and parts of Canada, these two regional agencies took no active steps to stop the progression, largely because they were unable to see the full extent of it.
This is partly true, though I'm not certain that MISO would have reacted if they could see. They are considered pretty inexperienced.
On Aug. 14, the trouble-shooters at Midwest I.S.O.'s nerve center in Carmel, Ind., spent much of the day coping with an earlier crisis in Indiana, and there are ample signs that their attention was distracted from the problems in Ohio, and that they were stretched too thin to handle both problems at once.
This is very likely - its normal to staff for normal workflows, but when things get wild, it becomes difficult if not impossible to keep up.
"It is an unacceptable level of confusion," said Ian A. Hiskens, a former electric utility executive and a professor of electrical and computer engineering at the University of Wisconsin, who ran PowerWorld simulations. "They should know what the state of their system was, that is fundamental to operating. And by 4:07 they are still not sure what has happened."
I agree here - there was confusion about what was happening, and there was probably confusion as to who had authority to do something about it, assuming anyone really knew that something needed to be done.
Coordinating the response to Cinergy's troubles involved at least six officials at Midwest I.S.O. They were hindered, too, by the failure of an I.S.O. computer program that left them unsure about what to do.
The program, called a state estimator, helps to monitor grid conditions and tries to predict what would happen if breakdowns were to occur.
"The state estimator has been down for an hour and a half," said Ron Benbow, lead monitor with the I.S.O. said at 2:36.
Here's where experience would play a key role - the state estimator is great for getting hardcopy to back up a decision, but an experienced operator should know the limitations of his system. That's assuming, of course, that real-time data is available. For Firstenergy, it was not. However, from what I saw in the transcript, MISO did have load data, and thus could have at least informed FE of what lines were carrying what loads.
This is assuming, of course, that there was a good perception that a major problem existed. I don't think the human side of this equation recognized the urgency - they relied too heavily on the computer's estimation model, which was not working.
- a new tool named for "Area Control Error (ACE) Frequency Real-Time Monitoring System" released this last year "designed to ensure the reliable supply of electricity" (so they say on their website) -
- on the MISO transcripts.
Judging from the MISO transcripts this tool too was not usable - there are several references to 'ACE' in the transcripts that I've yet to see mentioned in news/wire stories ... it's my contention, again, that due to network congestion that day data was not able to make it from point A to point B either at all or in a timely manner in order for these network stability and 'security' (not physical security in the usual sense but rather 'reliability and stability of the grid') tools/programs to function correctly and in a timely manner.
ACE, it appears, is centrally hosted and operators access info as needed - references on the transcript indicate trouble 'getting in'.
I need to go back into the transcripts and note where, what time, what context and exactly wheat the complaints were in accessing ACE ...