Also, I should add, there was discussion about "ACE" - - a new tool named for "Area Control Error (ACE) Frequency Real-Time Monitoring System" released this last year "designed to ensure the reliable supply of electricity" (so they say on their website) - - on the MISO transcripts.
Actually, ACE isn't really new - its essentially an equation that measures power flows into and out of a control area, balances them against area load, and applies a frequency component (60 cycles per second) and a mathematical constant that adjusts for the response rate of that individual control area's internal generation. It is a tool for each control area that allows them to maintain their generation to load balance.
Judging from the MISO transcripts this tool too was not usable - there are several references to 'ACE' in the transcripts that I've yet to see mentioned in news/wire stories ... it's my contention, again, that due to network congestion that day data was not able to make it from point A to point B either at all or in a timely manner in order for these network stability and 'security' (not physical security in the usual sense but rather 'reliability and stability of the grid') tools/programs to function correctly and in a timely manner.
Long before MISO, each utility took care of its own ACE. Today, it is still done that way, though MISO was possibly able to monitor the ACE of each utility control area within its region. I'll have to read some of the transcripts - while it isn't essential for MISO to view the ACE of each entity within, there still needs to be an ACE for each separate control area, and a record of how each utility maintained their ACE. Ours was once a mechanical chart recorder, but that has since been replaced by computer. There's supposed to be a backup scheme supplying all the necessary data to maintain control area integrity.
ACE, it appears, is centrally hosted and operators access info as needed - references on the transcript indicate trouble 'getting in'.
ACE is hosted centrally to each control area. I suspect that MISO wasn't able to read the ACE of one or more of the utilities in its area, probably due to computer problems. For example, if FirstEnergy's EMS computer was locked up, it was probably unable to supply ACE indication to its own operation center, let alone that of MISO.
BTW, I'm going out on a limb here by saying that MISO can or is even entitled to look at ACE for each utility - I'd probably better read some more transcript before I get too far down that road. Rules are different up there now.
I need to go back into the transcripts and note where, what time, what context and exactly wheat the complaints were in accessing ACE ...
Me too! Context can tell a pretty good story. Thanks.
BTW, thanks for the freepmail the other day. It is appreciated.
posted on 10/03/2003 1:28:01 PM PDT
I might should have linked to this: "a new real-time monitoring tool developed by CERTS". Their words describe this particular tool as 'new' - it looks like it's the aggregation of *all* the 'ACE data' from all 143 control areas in North America. There is also this +900 KB pdf file that goes into more detail.
This automated utility/program is, according to this diagram, hosted at NERC's Princeton, New Jersey headquarters and would, indeed give every utility/operator access to their neighbor's (and the entire grid's) 'performance'. Fan out to using entities followed this time frame:
ACE Completion and Delivery to 22 Reliability Authorities ? June, 2002
o ACE Users Training - July, 2002
o AIE Trial Test Starts - June, 2002
o AIE Final Delivery - September, 2002
(Sorry for the large image!)
posted on 10/03/2003 5:13:16 PM PDT
(Resources for Understanding the Blackout of 2003 - www.pserc.wisc.edu/Resources.htm)
I took the liberty of extracting those comments indicating any mention of tools/instruments involving *data communications* (including internet, RCIS and the State Estimator) as well as any tree-induced failure comments and placed them into one file.
- This file is meant to act as an index into the actual transcript files for further review of those areas of interest given this 'index' and it's subject
- This is only for pdf files a - d at the moment.
- Times are 7 minutes ahead of MISO network time (per the transcripts)
- Time is given in CDT (therefore the blackout ocurred at 15:09 vs 16:09 for EDT and would show as 15:16 in the transcripts due to the 7 minute offset)
- - - - - - - - - - - - - -
Pg 15 1347hrs
Ameren/Rick: This is Rick at Ameren. Are you having trouble with your Internet site, or do you know anything about it?
MISO/Tim Johns: Okay. That was a call saying they're having trouble with the website.
Pg 16 1348hrs
Wisconsin Electric Co./Blaine: Hey, we're having problems with our ICCS, and I guess we were wondering if your were encountering problems?
Pg 19 1426 hrs
MISO/Chris Sullen: Because I can't get back into the system, and the messaging system is down, amd so on and so forth.
Pg 21 1432 hrs
Speaker: I know you guys are - like your whatever, ICC link is down and everything with everybody.
Pg 25 1440 hrs
PJM/Roger Cohan: 345 line. That line is locked out. They believe it's a tree, and there's a bunch of people that are converging on that as we speak.
MISO/Tim Johns: Okay. Stewart to Atlanta is locked out.
MISO/Rich Cobb: The MECS. They got a 2300 instrantaneous ACE, plus. So you might give them a call.
Pg 44 1524 hrs
MISO/Rich Cobb: On the RCIS you could see it. It's the top 10 ACE value thing. Yeah, just go to the ACE and you can see it. 1527.7 is what they're doing right now. And MECS is actually negative again now, so FE, I think is where the problem is lying now, according to this thinh anyway.
MISO/Tim Johns: Yeah, and I can't get them sons of bitches to come up. Hang on. They're saying -- orginally we're showing we lost some load up your way. We can't --
MISO/Tim Johns: Hey, I'm showing FE's ACE as 1400 plus. I'm on the NERC site right now.
Pg 47 1527 hrs (after massive trip)
MECS/Jim: I can't get out onto A.R.S..
Pg 55 1531 hrs
MISO/Don Hunter: Everybody is protecting your ACE.
End of a.pdf
- - - - - - - - - - - - - - - - - - - -
Pg 17 (this comment was after blackout)
SIGECO/Bruce: And generation is responding by keeping our generation down below our ACE I guess to help the interconnect out.
Pg 31 2003-08-14 1010 hrs (Note: back before blackout again)
-Eastern Time Correction Call-
Pg 74 1223 hrs
... I'm not real familiar with some of these names that are showing up on our State Estimator here. ...
Cinergy/Spencer: I haven't heard a report on that. I know the helicopter should be flying it by now, but I haven't heard a report on it yet. ...
MISO/Rob Benbow: No real time overloads? I don't know if you have State Estimator, look at Post Contingencies. ...
- - - - - - - - - - - - - - - - - - - - - - - -
Pg 11 1546 hrs
MISO/Don Hunter: On the messaging and the RCIS if we can get through there.
Pg 13 1549 hrs
Cinergy/Doug: Okay, here's the -- here -- I just want to male sure you guys were aware that the 345 line that we lost was because of a tree inside the line.
Pg 24 1609 hrs
American Transmission Co./Bruce" I never did see it. And my SA wasn't working right - I had a guy work on it, and it is now.
Louisville Gas & Electricity/Tom: Well, we got a -- compared to New York and that area up there, we're in pretty good shape, but we do have a tree that caused aour 138 line to relay - it's done it twice. [Ohio County to Bonneville in Litchfield - the tree is between Shrewsbury to Bonneville]
Pg 57 1702 hrs
OATI/Nelson Mueller: Did you get it working?
MISO/Don Hunter: Still waiting, it takes a long time. Hey, I got to an acknowledge screen. Right on.
OATI/Nelson Hunter: It's just very slow today?
MISO/Don Hunter: Yes, it's very slow.
OATI/Nelson Mueller: More than it has been... I don't see the slowness, because I'm on the --
MISO/Don Hunter: You're right there.
OATI/Nelson Mueller: Yes, I'm on the local wetwork, so I don't see that.
MISO/Don Hunter: Yes. It's slow for me.
OATI/Nelson Mueller: Okay, but it hasn't been slower than normal?
MISO/Don Hunter: Yes, not this slow all day.
OATI/Nelson Mueller: Okay.
MISO/Don Hunter: It seems like it's starting to get slow here. It looks like it's working, and I'm not sure what's going on.
OATI/Nelson Mueller: Okay.
MISO/Don Hunter: All right, maybe it's nothing, maybe it's just me. I tried --
OATI/Nelson Mueller: There's a lot of activity going on right now that I see.
MISO/Don Hunter: Yes.
OATI/Nelson Mueller: So ...
MISO/Don Hunter: All right.
MISO/Don Hunter: I can't even freaking do anything. My IDC is so slow and crappy, it's unreal. I'm getting mad.
MISO/Don Hunter: No I haven't seen it yet. And why haven't I seen it it? Becasuse the internet's real slow right now.
First Energy/Joe: We don't have access to OATI or anything right at teh moment. Everything's supposed to be going to zero.
First Energy/Don: We'll start reloading them. But we have no internet connection to access OATI to do any of that stuff. So, we'll probably be working through you.
End of c.pdf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Pg 4 1347 hrs
MISO/Don Hunter: Hey, Dave. This state estimator has got some malfunctions happening here.
MISO/Dave Dieser: I am looking for --- we are looking at it right now.
Pg 5 1358 hrs
SPEAKER: The dynamic stability problem is ????.
Pg 26 1453 hrs
MECS/Jim: I am doing well. Hey, you guys experiencing any of your webpage problems or internet problems?
MISO/Don Hunter: I Heard there was a problem with some messaging. I have not had a chance to investigate yet.
MECS/Jim: Well, all my guys at scheduling stuff is all off-line here. I can't see to link it back up to you. I just wondered if you guys knew anything about that.
MISO/Don Hunter: I don't know anything about it out here. The schedulers are in the back though, and so they might know something about it.
MECS/Jim: Help me find something out and let me know will you?
Pg 32 1526 hrs
... I know that some of your control areas, you have seen your ACE's pick up, and that would be right. That would be a correct response for the bias, because that is going to make it a lower generation.
End of d.pdf
- - - -
posted on 10/03/2003 6:28:24 PM PDT
(<-- More resources/click on name ... also: Blackout of 2003 --> www.pserc.wisc.edu/Resources.htm)
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson