Computer failure may have caused D.C. train crash

Associated Press ^ | 6-23-2009 | AP

[normanpubbie]

WASHINGTON – Investigators looking into the deadly crash of two Metro transit trains focused Tuesday on why a computerized system failed to halt an oncoming train, and why the train failed to stop even though the emergency brake was pressed.

At the time of the crash, the train was also operating in automatic mode, meaning it was controlled primarily by computer. In that mode, the operator's main job is to open and close the doors and respond in case of an emergency.

Debbie Hersman, an investigator with the National Transportation Safety Board, said it was unclear if the emergency brake was actually engaged when Monday's crash occurred. But the mushroom-shaped button that activates it was found pushed down in the operator's compartment.

[normanpubbie]

If the emergency brake button was engaged, was the train still under automatic mode?

[pissant]

Computers failing should NEVER be enough to cause a train accident. What kind of piss poor design is this?

[Perdogg]

WMATA is always claiming they don’t have enough funds.

[pissant]

Bet they have plenty of useless bureaucrats though.

[newnhdad]

yeah, the driver’s i-pod was taking too long and she forgot to do something..

[ArtyFO]

Was the operator texting again?

[Sundog]

No, the mushroom shaped button was just an input to the computer.

[HangnJudge]

Computer failure
Hmmm...

I hear there are lots of industrious
Chinese programmers engaging in
Internet hacking

One wonders if someone got onto the
Computer servers running the subway system

Nah,
I would never happen

[Beelzebubba]

Blue Screen of Death?

[FreedomCalls]

If the emergency brake button was engaged, was the train still under automatic mode?

The button was probably pushed down because the operator was thrown against it after the train had already hit the one in front of it. Pushing the emergency brake button after the incident has already happened doesn't usually do much to stop the accident from happening in the first place. It does however act as a nice shield for the operator to hide behind and deflect blame.

[Bobkk47]

Subway buff info:

The Washington DC metro is computer controlled (as is the SF BART system). The train operator doesn’t normally “drive” the train. His/her job is to open and close the doors and to provide a “presence” in an emergency situation. On the older systems (NYC for example) the train operator does “drive” the train, although there is a “failsafe” signal system which can stop the train in the event the train operator becomes incapacitated (drops dead of a heart attack while the train is in motion etc.).

[Falcon4.0]

"Computers failing should NEVER be enough to cause a train accident. What kind of piss poor design is this?"

You are so right. When ever Human lives are at stake they're should be redundant systems. Just like in Aircraft.

[Cboldt]

-- the mushroom shaped button was just an input to the computer --

Heh. If so, that's a fundamental design error, forbidden by ALL written codes for design and construction of control systems. Emergency anything is supposed to operate whether or not the computer is even there.

[Cboldt]

-- When ever Human lives are at stake they're should be redundant systems. --

Yep. When the downside risk is serious personal injury or death, common design practice includes redundancy; where the redundancy is outside the computer.

For example, if safety is obtained by opening a power circuit, then redundancy is two sets of power contacts in series, so if one sticks closed, the other opens the circuit.

[La Enchiladita]

Was the operator texting again?

My guess is yes. Investigators will be looking at cell phone and texting records.

[La Enchiladita]

I encountered conflicting reports on this in this Baltimore Sun article (via L.A. Times): 9 fatalities confirmed in D.C. Metro train crash

One DC Metro source says the trains run on automatic, another says the trains were all being operated manually at that time.

[wrench]

If this event wasn’t hacker/terrorist, the next one certainly will be.

Now that a weakness has been revealed, the hackers will exploit it.

[rabidralph]

Piss-poor design is Washington, D.C.’s middle name.

[rabidralph]

They probably had one of these installed in its place.

[Right Wing Assault]

They said there was heat discoloration on the rotors indicating the the brakes were activated at some point. When and how long is another story.

[Tolerance Sucks Rocks]

Maryland “Freak State” PING!

[normanpubbie]

Here are two articles from the WaPo which give the most detail of any I've found:

At Least 6 Killed in Red Line Crash

THE PROBE: Experts suspect failure of signal system, operator

The operator was probably killed instantly. An earlier report asserted that the recovery people had found bodies in the compressed part of the first car.

Here's the deal on the "mushroom" or emergency stop button which apparently was engaged at some point. Emergency stop buttons on vehicles are called "mushrooms" because that's what they look like. They are typically 2 to 3 inches in diameter and built like a tank. The switch normally stick out. You hit the mushroom knob and the switch latches into place. Some models are reset by turning the knob and others have key-operated resets. The mushroom is connected directly to the power system and braking system, probably through a relay. It will not be connected through the computer.

Debbie Hersman is the National Transportation Safety Board member in charge of the NTSB investigation.

Hersman said it wasn't clear when the button was pressed or how it got that way. She also said there was evidence of braking on the train's rotors, indicating it was likely that the operator tried to slow down.

Apparently we are supposed to infer from this statement that the operator pushed the button and that the brakes were engaged.

Color me skeptical. There is no way that the investigators have inspected the brake system this early. And the "mushroom" could very well have been pushed in when the cab pancaked during the collision.

Just look at the pictures. It's hard to believe that the brakes had been applied before the crash. And no one can say that any wear present on the rotors or track occurred immediately before the collision.

My theory is that the operator was running the train in manual mode and that she never applied the brakes. None of the people on the rear train have said that the train braked just before the crash. Even the people in the last car would have felt braking. And it's hard to believe that, if the train had been on automatic control, that all of the automatic safety systems failed simultaneously.

[FreedomCalls]

They said there was heat discoloration on the rotors indicating the the brakes were activated at some point. When and how long is another story.

One can assume that the brakes are activated on a regular basis throughout the day as a matter of routine.

[Sundog]

I like that.

What is the meaning of life?

[Right Wing Assault]

activated on a regular basis throughout the day as a matter of routine

But I don't think routine braking would cause the heating needed to discolor the metal.

[the OlLine Rebel]

It’s possible braking happened at the last second (woman couldn’t see there was any obstruction until too late? - depends on environment).

Most assuredly heat discoloration on the rotors is NOT going to occur if the train already crashed (i.e., the woman’s body was thrown against the button), because the train is already “stopped”.

[the OlLine Rebel]

“Just like in Aircraft.”

Of course, there are occasions when that’s not necessarily good. Recall Payne Stewart’s plane, where all were mysteriously passed out and the plane was just ambling around for hours.

[normanpubbie]

This event illustrates perfectly my beef with the NTSB. The NTSB always comes up with a "probable" reason for an accident immediately, before any serious investigative work hs been done. In this case Hersman made four statements that the agency will eventually have to walk back:

1. Some fatalities and injuries were caused because the rear train was not "beefed up" as the NTSB had previously recommended.
2. The train was on automatic control at the time of the accident
3. The operator applied the brakes before the crash
4. There was wear, discoloration, etc. on the track and rotors

On point 1, you are talking about two vehicles which weigh many tons each. The rear train would have enormous kinetic energy travelling (presumably) 55 MPH; there would be massive damage regardless of the construction of either vehicle. But the deaths and injuries were caused by the collision, not by shoddy construction. Flaky logic on the part of the "bigger government, bigger budget" folks.

On point 2, maybe the switch in the control center was on automatic, but that doesn't mean that the one in the cab was or that the train was actually on automatic.

On point 3, look at the pictures linked in #16. Can you still insist that the brakes had been applied before the crash?

On point 4, maybe last month the driver hit the emergency stop to avoid hitting a stray animal on the tracks, then released the stop and continued on. That would cause discoloration of the rotors. Would the driver have reported it? Maybe, maybe not.

In this morning's reports, the authorities are saying that the train was on automatic control and that the brakes had been applied. That is contradictory.

My scenario from #22 fits the situation on the ground a lot better than Hersman's politically-correct "facts."

[Hulka]

With a request for $1.4B for their budget, pleading poverty doesn't work and their incompetence and corruption likely played a role in the mishap, as they squandered funds and stuffed their pockets:
http://www.washingtonexaminer.com/opinion/Metros-deadliest-crash-and-the-culture-of-secrecy.html

Metro's deadliest crash and the culture of secrecy
By: Examiner Editorial

06/24/09 5:19 AM EDT If you were in Buenos Aires, Argentina, or Ottawa, Canada, at 6:30 pm Monday evening, you may well have already heard about the deadliest accident in Metrorail’s history.

But if you were in a Metro station here in the D.C. area, all you heard on the system's public address system was something vague about “a train experiencing mechanical difficulties outside Fort Totten station.” In other words, for hours after its deadliest crash ever, Metro kept its riders in the dark about what had happened. Metro has done the same thing for years concerning how it spends its millions of dollars in subsidies from taxpayers.

Secretive cultures tend to produce such closely related phenomena. We may not know for another year why Monday's tragedy happened, but what we do know now is that for years Metro has delayed critically important maintenance, even as it spent exorbitantly on grandiose expansion plans and excessive employee benefits. We also know that Metro neglected safety upgrades recommended by the federal government three years ago to the same model of rail car that performed so poorly in Monday's crash. After a 2004 train crash injured 20 passengers at the Woodley Park station, the National Transportation Safety Board (NTSB) recommended crash-worthiness upgrades to Metro's oldest rail cars, its original 1000 Series, to prevent “a catastrophic compromise of the occupant survival space.” That Metro failed to make these upgrades may not have caused this week's crash, but it might have made it more deadly.

A mangled 1000 Series car is the one seen stacked atop the train it had hit from behind. Metro officials likely will soon demand more tax dollars to make its system “safer.” But what about the billions already paid in fares and taxpayer subsidies? Metro paid six-figure salaries to more than 5 percent of its hourly workforce in 2006. The same year, Metro also paid $70 million in overtime, an average of $41,000 to its 408 best-compensated bus drivers, train operators, mechanics, and the like.

We would publish more up-to-date figures but Metro stopped sharing such data with The Examiner after we exposed the system's compensation practices. Before Monday's disaster, four Metro employees were killed by trains between October 2005 and November 2006, in three separate incidents involving mechanical or human error. Something is terribly wrong within Metro, but it won't get better as long as the accountability that comes with sunlight is kept out.

[Hulka]

I ride the Metro every day.

Operators are barely literate and shut the doors on passengers, and I mean shut the door ON passengers (the doors are not like elevator doors with an auto-open if someone is trapped). I predict there will be a mishap where an operator closes the door and traps a passenger in the door, half in and half out. Count on it.

[Hulka]

I read this morning in the DCExaminer paper that there were nine fatalities, but on-line for the paper it shows eight: http://www.washingtonexaminer.com/local/Metro-officials-began-releasing-the-names-of-people-who-died-in-Mondays-crash-48904417.html

[La Enchiladita]

My theory is that the operator was running the train in manual mode and that she never applied the brakes.

I'm with you. I wonder if we'll ever learn the reason she did not apply the brakes. For the disastrous train collision here in L.A. area last year, we learned the train operator was wildly texting.

[the OlLine Rebel]

“On point 3, look at the pictures linked in #16. Can you still insist that the brakes had been applied before the crash?”

What has that to do with anything? One cannot tell a single thing from those pix (and I scrolled through ALL 48 of them, hoping for a pix of brakes).

“On point 4, maybe last month the driver hit the emergency stop to avoid hitting a stray animal on the tracks, then released the stop and continued on. That would cause discoloration of the rotors. Would the driver have reported it? Maybe, maybe not.”

I seriously doubt an operator would slam on the brakes from a top speed for an animal. That would’ve caused people to plunge out of their seats (no seatbelts), never mind people standing. 1st of all, top speed isn’t often reached due to tight areas, and 2nd, the chances of a major animal (deer)happening to show up on the tracks in all the miles of Metro is somewhat iffy.

Look at the obvious curve on that stretch, complete with some trees at some part of it. Given that (and nothing else, because there aren’t better pix), it’s very possible this woman was at top speed and saw the stationary train at the last couple seconds.

Is it possible all the worst happened? Sure. But I’m not hopping on the worst-possible-negative-viewpoint bandwagon based on what we do see.

[Hulka]

Based upon my decade-plus riding the Metro, with the last 4-yrs riding on my daily commute, in my jaded view I think incompetence first, then maintenance second as playing a primary role in the mishap.

[jaydubya2]

Using a computer input as an Emegency stop is a big NO NO. The E-stops I am familiar with are typically hardwired directly to the energy source. That said, sometimes a device can be stopped at a faster rate if energy is availible. For example, a motor can be stopped faster if energy is available to dynamic brake the motor or energize a mechanical brake (if the brake is not a failsafe design).

Under NFPA 79 2007, it:

Shall override all other functions and operations in all modes, and

Power to the machine actuators that can cause a hazardous condition shall be removed as quickly as possible without creating other hazards (e.g., by the provisions of mechanical means of stopping requiring no external power)

Reset shall not initiate a restart

[Cboldt]

-- That said, sometimes a device can be stopped at a faster rate if energy is availible. --

Much faster, in fact. Printing presses will coast for 5-10 minutes but can be stopped in seconds under dynamic braking. And then there are facilities that become dangerous if power is removed, e.g., magnetic workholding.

[the OlLine Rebel]

Could be.

I rode for about a year back in ‘94. It was a good system. But I guess the people don’t have to be genuises - or workaholics.

My biggest incident for my commute was - indirectly - on the MARC train. A MARC commuter woman was struck by a freight train at Laurel (1 of my commute “buddies” on both MARC and Metro sort of witnessed it - she had been right in front of him walking to the platform) - for which we are forever seeing those cute little yellow signs on blue poles at ever RR station crossing in MD.

[the OlLine Rebel]

I think that statement was wrong. I read - either a poster in the know or an article - it has nothing to do with CPUs and more just with electrical power. Which to me is also questionable.

[normanpubbie]

#33: Fair points. Here are my replies:

1. The reason there are no pictures of rotors is that they are behind the wheels -- some disassembly is required to inspect them, and that sure didn't happen by the time the alleged rotor damage was reported.
2. OK, an operator would not hit the brakes hard to avoid an animal standing on the tracks. But would he/she hit the emergency brake to avoid hitting a transient on the tracks or a car stuck at a crossing? You betcha.
3. Yes, the train was on a sharp curve but this does not mean that there wasn't a good line-of-sight to spot the stopped train.

Why not think the worst? Nine people died so far. The train was driven by an inexperienced operator (18th from the bottom in a group of 523 operators). According to media reports, Jeanice McMillan joined Metro in January 2007 and was required to serve a year as a bus driver before taking the 12-week course to be a train driver. That's 15 months minimum. But she became a train operator in December 2008, just 11 months later. Please explain that inconsisency. Better yet, ask Metro management to explain it.

Hersman has stated that there was heat damage to the rotors and physical wear to the trackage, implying that both occurred in the accident.

This is logically inconsistent. If the wheels continued to turn at the speed that the train was travelling (i.e., no slippage), you would have overheating (blueing) of the rotors but no track or wheel damage. On the other hand, if the wheels were slipping with relation to the track (as would happen if the rotors were locked or nearly locked), you would see a lot of mechanical damage -- especially on the wheels but also on the trackage. When the wheels are slipping the stopping power is virtually nil. And no braking was reported by any passenger quoted so far.

Someone is at fault here, and we need to rule out operator error before ruling in other stuff like automation failures, signalling "anomalies," brakes were applied but didn't stop the train, etc., etc. as the authorities have done up to now.

And in almost all cases mass transportation accidents are eventually found to be due to human error.

#35, Point well-taken, Cboldt. I'm sure that the emergency stop process employs both dynamic braking and mechanical friction.

[Cboldt]

-- Someone is at fault here, and we need to rule out operator error before ruling in other stuff like automation failures, signalling "anomalies," brakes were applied but didn't stop the train, etc., etc. as the authorities have done up to now. --

No doubt a combination of factors. The DC Metro system had a reported incident years ago where the automatic signaling failed, and the operator prevented a collision. The car manufacturer (who is certainly going to be sued, as will all the various designer/component makers) will assert that one function of the operator is to override automatic operation, when it is obvious the automatic system is giving and/or acting on erroneous information. E.g., if you see you are moving, but the speedometer reports -zero-, then you should use the brakes to slow the vehicle.

Under normal operation, the only nominal function of the operator is to open the doors. Under abnormal operation (signaling errors), the operator can control the car/train.

Some Technical Information on the Design

All Metro cars are equipped with both dynamic (electronic) and friction braking systems; you can hear the "whine" of the dynamic brakes as the train comes to a stop. The overall system is known as Automatic Train Control (ATC), and controls all train movements - braking, acceleration and speed control, but can be manually overridden by the attendant in the case of an emergency. In each cab is a fully digital console, with all of the appropriate controls. The train's maximum allowed and current operating speeds are shown via orange (Rohr) or red (Breda) LEDs and are visible from behind the cab window. There is also a master train control handle as well as an emergency stop button on the console.

[the OlLine Rebel]

Bottom line is, sounds you are taking the words of the NTSB people and associated too seriously. They didn’t say they did an investigation; it’s just cursory rudimentary observation.

[Cboldt]

DC Metro Accidents

On January 6, 1996, during the Blizzard of 1996, a train operator was killed when a train overran the Shady Grove station and struck a parked train. It was later determined that because operators tended to overuse braking systems and wear them down, only computer-controlled braking was allowed to be used by operators. This operator had asked to be permitted manual control over braking and was refused permission only a few minutes before the computer-controlled braking system failed to stop his train in time. An NTSB investigation found the following factors that contributed to the accident: At the time of the accident, there was a policy then in effect that prohibited supervisors from granting employees permission to operate trains manually, even in inclement weather.

This incident is going to have a complex answer as well.

[normanpubbie]

Re #42: The NTSB also faulted management for parking a train on the active track rather than on a side track.

The operator of the moving train asked for manual control of the train because he knew that the automatic system was unable to stop the train in the snowstorm.

So the 1996 accident had two root causes: Parking a train on the main line and also denying permission for manual control which was prudent under the extraordinary weather conditions. That's not complex -- that's human error times two.

There was another near-tragedy in 2005 (citing my original post on Yahoo which has gone down the memory hole but is still available at other places on the Web):

In June 2005, Metro had a close call because of signal troubles in a tunnel under the Potomac River. A conductor noticed he was getting too close to the train ahead of him even though the system indicated the track was clear. He hit the emergency brake in time, as did the operator of a train behind him.

The operator of the middle train realized that the auto system was not functioning properly and hit the brakes. So did the operator of the rear train.
Result: No accident.
Conclusions: Both operators knew what they were doing and were paying attention.

By the way, according to the Hersman's page on the NTSB website, her appointment as a member expires December 31, 2008. According to other news accounts, Barack Obama appointed her chairman of the NTSB. The NTSB press release page, which is up-to-date, identifies Mark Rosenker, appointed by George Bush, as the acting chairman. It may be that Obama appointed her but she has not been confirmed yet and that is why Rosenker is the acting chairman.