Calls Show Pre-Blackout Utility Confusion

AP ^ | 9/3/03 | JOSEF HEBERT

[Pro-Bush]

engineers in the control center of an Ohio utility struggled to figure out why transmission lines were failing and complained that a computer failure was making it difficult to determine what was going on

Hmmm.

[Russian Sage]

Virus, Worm or just your everyday Microsoft pain?

[Mind-numbed Robot]

Virus, Worm or just your everyday Microsoft pain?

Blue screen of death. Please reinstall Windows.

[Crusader21stCentury]

Virus, Worm or just your everyday Microsoft pain?

Try this on for size... The guy who knew what he was doing asked for a raise and was told to go to hell. He left for greener pastures and the entire east coast goes down. I'll bet this isn't far from the truth.

[Milwaukee_Guy]

Still amazed that the cause of the blackout hasn't come out. If the operators were actually in control they would know what had happened in real time.

Now it's just a face saving farce for all involved.

[Raymond Hendrix]

Ah yes, the computer age. This reminds me of an airliner that was flown into the bay in San Francisco several years ago because the computer decided that was the best thing to do. Cashiers cannot even make change now day if the computer goes down. And to think we are all at the mercy of one 14 year old kid...

[lelio]

Or they 'outsourced' their IT department to BigMultinationalCorporation that has a turnaround time of 2-3 days for requests.

[.cnI redruM]

Microsoft sux sometimes, but sometimes the Gates haters remind of Bush haters in their irrationality.

[mamelukesabre]

It seems to me that most people would rather do a task, any task, via computer (or high tech gadget) even when doing it the old fashioned way is just as efficient or more so.

here's a very very mundane example...

Those catsup dispensers at burger king...they used to be a manual operated submerged pump. all you needed to do was push down on the plunger one time and the appropriate amount is dispensed in your paper cup. But no more! Oh no! Some genius decided that wasn't good enough. Now we have a pneumatic pump system that squirts catsup through a tiny little orifice. To make matters worse, there is a time delay from when you depress the electric switch to the time there is enough pressure(or is it vacuum?) built up to get the catsup moving through that rediculous little orifice. Then, once it starts coming out, its coming out at warp 5....and naturally, there is a second time delay from when you release the electric button to the time the catsup actually stops squirting. It is virtually impossible to get your paper cup filled just the right amount...AND, it takes 3 times longer to do a poorer job of it. How's that for technology?

Now, let me recap:

The old system was cheap, simple, cannot break down, doesn't require power, worked quickly, and flawlessly and even a pet raccoon has the hand-eye coordination to work it properly.

The new system costs more, and replaced a system that was already in place and functioning and it is unlikely that any salvage value was gotten out of the old system...it was likely just thrown away. The new system is complicated, requires power, makes messes, frustrates the cusomers, and the line at the catsup dispenser now tends to be longer.

Who the * actually thinks this is an improvement!!

[meyer]

Well, they spelled Jerry's name wrong, but things are starting to come together.

The computer issue might make one go "hmmm", except that this particular computer that FE still uses has been known to be down for long periods of time. If I could describe it in two words, it would be "self-hacking". Until 2 years ago, I worked in that very control center, and I remember it being unreliable. I also remember it giving all kinds of meaningless alarms to the point that we were stressed to dig out the important alarms in a sea of useless information. Our complaints were ignored. I suspect complaints will no longer be ignored on this matter.

I have some additional information from a couple of the folks up there in Akron/Cleveland. It seems that the reason that Eastlake unit 5 tripped off was instability in the exciter. The unit was having trouble regulating voltage due to the system voltage sagging. The exciter overheated and tripped the unit off. Its not unusual when transmission loading is very high to have voltage problems in that part of the grid, particularly since several of the smaller, older generators along Lake Erie have been decomissioned along with Davis-Besse Nuke still being off line. There's not a lot of generation up there to help the system out.

Also, my conversation yesterday revealed that there were very large power flows into Michigan that afternoon. My friend tells me that the flow through FE's system that afternoon was around 1500 MW, which is fairly substantial. I don't know if the midwest ISO was allowing this flow, or if it was being done "off the books". There's a lot of trust among utilities that no longer exists in the competitive market and as a result, there are likely to be side transactions that would never be allowed if NERC policy were being followed. I have no proof of this, but I'm highly suspicious, given the financial rewards for "cheating" and the lack of punishment for doing the same.

[meyer]

Virus, Worm or just your everyday Microsoft pain?

BTW, FE's energy management computer uses proprietary software designed by GE-Harris, built on a UNIX platform. No Microsoft there. The system was also designed to operate a grid about 1/3 the size of FE's grid.

[mewzilla]

Just out of curiosity, am I correct in thinking that First Energy is a Canadian firm?

[chimera]

BTW, FE's energy management computer uses proprietary software designed by GE-Harris, built on a UNIX platform.

I am forced to use GE's on-line invoicing system, such as it is. I have a strong suspicion that this system was outsourced to New Delhi. When you call their help desk (which is frequent given the problems it has) you get someone who speaks in Indian-accented broken English, who tells you he doesn't know what the problem is and you're to call another number. So I'm wondering if their grid management software, if it comes from GE, might not have been similarly outsourced.

If so, it shows you the very, very serious downside of overseasing important software. Save a few pennies in the quarterly bottom line by firing your in-house IT dept., and take down the power system for half the country's population.

[winodog]

If I uninstall windows and reinstall, will I lose other things such as quickbooks, office jet printer, etc?
Would I unstall all things labeled microsoft including outlook express and internet expolerer?
I deleted something by accident and my mcafee firewall will not reinstall till I fix microsoft.
Any help would be appreciated. Thanks

[meyer]

I am forced to use GE's on-line invoicing system, such as it is. I have a strong suspicion that this system was outsourced to New Delhi. When you call their help desk (which is frequent given the problems it has) you get someone who speaks in Indian-accented broken English, who tells you he doesn't know what the problem is and you're to call another number. So I'm wondering if their grid management software, if it comes from GE, might not have been similarly outsourced.

That may well be, but it wasn't the case (I don't think) in 1999 when I was still there - GE sent engineers out on a few occasions in the late 1990's to help tame the system. Most of them came from the west coast. I don't think they ever sent the same engineer twice, due to turnover in the industry at the time. Lots of job-hopping, along with some crappy software apparently.

Even if the system worked perfectly, its design and setup was not user-friendly. The power grid layout was basically a giant "Auto-Cad" type of print that required a lot of scrolling around and zooming in and out just to follow a line to its destination and read its loading. The alarming priorities were all askew, with useless information flooding the screen that should have been reserved for only the most important data. The alarm bell rang for everything from lines tripping to minor overloads to substation doors being opened and closed. It was not designed with the operation of a grid in mind. The problem was that the in-house IT department dictated what the users got instead of the users dictating what the IT department should give them.

[meyer]

Try this on for size... The guy who knew what he was doing asked for a raise and was told to go to hell. He left for greener pastures and the entire east coast goes down. I'll bet this isn't far from the truth.

That's plausable, given the way they ran the place after the merger. Frankly, I left 2 1/2 years ago and not necessarily for a raise - the corporate "culture" (and I hate using that left-wing phrase, but it applies) just plain stunk there.

[chimera]

I wrote up some training documents in the mid-1980's for GE's TRA (Transient Recording and Analysis) system. It was poorly documented and the code "looked" flaky when it ran. The module I worked on, which was the reactor stability analysis, used a time series approach and eventually did a Fourier transform to frequency space, which is a standard method of noise analysis, but you have to be careful that normal fluctuations in the signals aren't misinterpreted as instabilities. Their system was very sensitive to that. I'm wondering now if this isn't a problem that might end up in GE's lap.

[meyer]

I wrote up some training documents in the mid-1980's for GE's TRA (Transient Recording and Analysis) system. It was poorly documented and the code "looked" flaky when it ran. The module I worked on, which was the reactor stability analysis, used a time series approach and eventually did a Fourier transform to frequency space, which is a standard method of noise analysis, but you have to be careful that normal fluctuations in the signals aren't misinterpreted as instabilities. Their system was very sensitive to that. I'm wondering now if this isn't a problem that might end up in GE's lap.

I'm sure the warranty is up on that system. :) Actually, I can only report as an end user - I am not well versed in software design. I can say with certainty, however, that the old IT folks from Cleveland were more responsive to dispatcher needs. They also squeezed over 20 years of service from an old Sigma-5 computer system, finally retiring it around 1994 or so in favor of an ABB design that was rejected at the merger (as were all things related to Cleveland).

Seriously, I think there is going to be a multitude of sins that occurred to set up this situation in the first place, but the lack of information provided by FirstEnergy's EMS computer precluded any reaction by system dispatchers (and given the top-down management of FE, I suspect that they wouldn't react without permission anyway, even though NERC policy empowers them to do so).

The midwest ISO had duplicative information as well through a separate system, or they should have. I recall an early article that indicated that they didn't think a single 345,000 volt line tripping in the Cleveland area was significant. Having operated the grid in the Cleveland area prior to the FirstEnergy merger, I would disagree. I think that the ISO was woefully unfamiliar with local constraints and thus didn't "see the trees for the forest" as it were. Their size precludes them from seeing important details.

[Fellow Traveler]

Sir,
If you have any other information on the exciter problems at Eastlake I would be very interested and I might be able to shed some light on what it means. I was a GE generator and excitation specialist for 6 years. I am familiar with most types of GE exciters. At first glance it appears that the part that is over heating is the generator field. Usually field temperature is inferred from a measurment of the field resistance. If the system voltage was sagging the generator would be trying to push high amounts of vars into the system and thus the field current would be high. High field current will make the field heat up and it could force a trip or curtailment.

[meyer]

Sir, If you have any other information on the exciter problems at Eastlake I would be very interested and I might be able to shed some light on what it means. I was a GE generator and excitation specialist for 6 years. I am familiar with most types of GE exciters. At first glance it appears that the part that is over heating is the generator field. Usually field temperature is inferred from a measurment of the field resistance. If the system voltage was sagging the generator would be trying to push high amounts of vars into the system and thus the field current would be high. High field current will make the field heat up and it could force a trip or curtailment.

I don't have direct information from the plant personnel, but considering Cleveland's constraints, lack of generation near the city, and the rather large power flows that were apparently going up towards Michigan from the south (I wonder who, if anybody approved that transaction), I would bet they were pushing all the VARS that the unit could supply. We often pushed the units pretty hard for both VAR and MW output in the summer months up there (actually, that's pretty universal in the industry). It isn't unusual to run close to the edge, but they apparently went over that line.

Someone I had talked to mentioned voltage fluctuations during the afternoon as well. They mentioned that a sudden dip may have caused the voltage regulator on the unit to push beyond the limit, either overheating the field or causing an overcurrent condition on the exciter. It may even have failed under stress. I'll try to get more information when possible.