How We Nearly Lost Discovery

Wayne Hale's Blog ^ | 18 April 2012 | Wayne Hale

[Yo-Yo]

Now that Discovery is safely delivered to the Smithsonian, I think I can tell the story of how we nearly lost her in July of 2005, and how well intentioned, highly motivated, hard working, smart people can miss the most obvious.

It’s tough to know people who have died. Many of us knew the astronauts on Challenger and Columbia well. We had met with them daily, we had visited in their homes, we knew their families, their children. It is not an easy thing to lose a colleague; especially one who entrusted their safety to you. So don’t question whether we were motivated to prevent another loss.

Discovery was the shuttle return to flight vehicle after the Challenger was lost; two and a half years were spent from January 28, 1986 until Discovery flew in September 1988. Many improvements were made which resulted in a safe space flight.

Discover was the shuttle return to flight vehicle after Columbia was lost; two and half years were spent from February 1, 2003 until Discovery flew in July 2005. Many improvements were made but safety was not assured.

It was not until Discovery again flew in July of 2006 before she flew safely. That counts as the third “return to flight” mission for Discovery.

You see, we dodged a bullet in 2005. One we should have seen coming but didn’t.

After the loss of Columbia, it became very clear that that the loss of insulating foam off of the external tank caused damage to the heat shield which lead to the loss of the good ship and crew. It was a devastating time. Tens of thousands of experiments, tests, and analyses were performed to discover why foam was lost from the external tank and how to prevent it. One of the most powerful series of test performed used a partial panel, about 2 feet square, of the aluminum skin of the tank covered with foam. This test article was chilled on one side to the cold temperature of the cryogenic hydrogen fuel, and heated on the other side with flames simulating the heat generated by supersonic flight through the lower atmosphere. Adding high velocity airflow to mimic the high speed flight conditions, virtually all the time the foam stayed on the tank. Only when small defects were embedded in the foam would some of that foam pop out under those strenuous conditions.

All the tests and analyses lead to the inexorable conclusion that defects in the application of the foam insulation could cause foam debris to be generated during the early supersonic phase of shuttle flight. We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia and then worked them overtime in training with new and exhaustive techniques on how to apply foam with no defects.

Many other safety measures were incorporated during those two and a half years; a new inspection boom with camera was built so that complete inspection could be made before shuttle re-entry; heatshield repair materials were tested and proven to work in the vacuum of space if repairs were needed, plans for safe haven at the International Space Station were formulated so the crew would be safe even if extreme, unrepairable damage were detected. Anybody who followed the shuttle program in those days knows all this.

What you probably don’t know is that a side note in a final briefing before Discovery’s flight pointed out that the large chunk of foam that brought down Columbia could not have been liberated from an internal installation defect. Hmm. After 26 months of work, nobody knew how to address that little statement. Of course we had fixed everything. What else could there be? What else could we do? We were exhausted with study, test, redesign. We decided to fly.

The launch seemed very normal, we had new cameras everywhere including one on the external tank itself. We all breathed a sigh of relieve when Discovery reached orbit seemingly without incident. The members of the Mission Management Team received an early briefing from the video review of no incidents; we boarded our planes from KSC to head back home, that mean Houston for me.

The next morning there was an early MMT briefing from the launch video review team, again nothing major to report; the crew had a busy day on orbit, my assignment was to go to the MMT press briefing where we gave a positive report. It was on the drive home looking forward to dinner and sleep when I got the call.

I think that must have been the worst call of my life. Once earlier I had gotten a call that my child had been in an auto accident and was being taken to the hospital in an ambulance. That was a bad call. This was worse. Think of the worst phone call you have ever gotten. I think this one was worse.

John Muratore, my good friend, fellow flight director, and current head of the shuttle program Systems Engineering and Integration office informed me in very flat terms that he was in the JSC video lab with head photo interpreter Cindy Conrad who had uncovered evidence of a large foam liberation during the critical mach number regime which appeared to have impacted the left wing of Discovery. Just like Columbia.

I made a highly illegal U-turn in the middle of NASA Road 1 and probably exceeded the posted speed limit heading back to JSC and the photo lab. Here is one still frame from the video they showed me: A very large piece of foam coming off the tank heading for the wing.

I thought I would not be able to breath again.

I thought of Eileen and her crew and how we had probably just killed them.

I thought of all the work and time and how wasted it had been.

I had no idea how I could possibly tell the team, the world, what had transpired.

But unlike Columbia, this time the crew had the means to inspect their heat shield. That inspection revealed no damage. The pictures and engineering data showed no damage. Positive and unambiguous evidence showed the heat shield intact and safe.

Then and only then could I start breathing again.

The next day we told the crew, sent them the video, told the entire NASA senior management, and at the press conference showed the video and made the characterization that this was “unsatisfactory”. A pretty bland word for the way I really felt

So what happened? The video wasn’t really clear so either the foam missed the underside of the orbiter completely or struck such a glancing blow that the impact resulted in no damage.

But that is not the real question to be answered. What really happened to the foam? We wouldn’t find out until December.

God gave us a great gift during the previous year. We had ECO sensor problems. At the time that did not look like a gift. We filled the external tank with cryogenic fuel as a test in May and the Engine Cutoff Sensors – little wet/dry indicators at the bottom of the tank – malfunctioned; indicated dry when they were wet. We scrubbed that tank from the first flight and sent it back to the factory to analyze sensors.

Now an external tank is a fragile and valuable thing. On a good day the cost estimate for a completed, flight ready tank was $30 million. On a bad day, more than twice that. We never, ever, had shipped a tank back to the factory. We tested them at the launch site, made occasional repairs, took lots of measurements, but never had shipped one back before that tank.

When we shipped it back – did I forget to mention that Hurricane Katrina devastated New Orleans the month after we launched Discovery? Did I forget to mention that the steering mechanism on the trailer the tank was on failed so we sent another tank back to Michoud before the critical one? Months passed and right before Christmas, we got these X-rays from Michoud.

They showed cracks in the foam. Foam where no defects existed. It turns out that the thermal cycles associated with filling the tank could crack the foam, especially in areas where there were two or more layers of foam.

Finally, this explained the Columbia foam loss. And the Discovery foam loss. And it had nothing to do with the improper installation of the foam.

I flew to New Orleans the next day, and called an all hands meeting where I publicly apologized to the foam technicians. They had not caused the loss of Columbia through poor workmanship. Those guys were reeling from the hurricane’s devastation to their homes and community, and has lived with nearly 3 years of blame. Thin comfort for me to apologize, so late, so little.

We worked feverishly to remove foam on foam wherever we could, minimize it where it could not be eliminated, and the following July we were ready to try again.

Discovery flew on July 4, 2006; no significant foam loss occurred.

I consider that to be the real return to flight for the space shuttle.

So were we stupid? Yes. Can you learn from our mistake? I hope so.

So when you go to the Smithsonian and see Discovery there, think how lucky you are to see her whole, intact, and with her crews safely on the ground.

You see, this is how I found out that we were never really as smart as we thought we were.

Maybe that is a lesson that applies to you, too.

[Yo-Yo]

So much for the "Green Foam Formula" theory...

[Ingtar]

Had they left the asbestos in, even foam on foam would not have been an issue - or so I have heard.

[ThomasThomas]

They could could still loose it by parking in DC overnight.

[Bogey78O]

Almost the same thinking messed up the Hubble. They were so sure the final tests done on it were off due to gravity warping the lens during the test that they made a simple mistake.

Some times you make mistakes even when you’re smart. You can’t assume you’re right because you’re smart.

[BitWielder1]

Why fragile foam on a flight surface?
Why not better insulation *under* the surface?
They could have used the same material as the re-entry heat shield tiles, they insulate well and are light weight.
Well, it's moot now.

[Red Steel]

They showed cracks in the foam. Foam where no defects existed. It turns out that the thermal cycles associated with filling the tank could crack the foam, especially in areas where there were two or more layers of foam.

It's amazing they missed this. I thought of this as a pssiblity before I read the paragraph.

[petro45acp]

Does that mean the there was enough dimensional change in the diameter of the tank through heat and cold caused the foam to de-laminate?

[GoDuke]

Then we shut it down 3 years later.

[June2]

This is off topic, but I’ve noticed this happen before. This topic “How we nearly lost Discovery” and the next article “Obama lawyer Angelo Genova tries to suppress & intimidate NJ conservative press & citizen journalist” don’t show up when I tried to look at the Discovery article on my Iphone. The articles above it “ You thought you knew Prostitutes.....” and the one below it “Swedish Stonehenge...” are there but the other two are not.

Why would the computer Freerepublic be different from the Iphone?

[Jack Hydrazine]

I thought the ET foam performed better with the freon-based foam than the new environmentally-friendly foam.

Shuttle Foam Loss Linked to EPA Regs
http://archive.newsmax.com/archives/ic/2005/7/28/93055.shtml
2005

(excerpt)
As recently as last month, NASA had been warned that foam insulation on the space shuttle’s external fuel tank could sheer off as it did in the 2003 Columbia disaster - a problem that has plagued space shuttle flights since NASA switched to a non-Freon-based type of foam insulation to comply with Clinton administration Environmental Protection Agency regulations.

Did PC Science Cause Shuttle Disaster?
http://www.foxnews.com/story/0,2933,77832,00.html
2003

(excerpt)
NASA is reconsidering whether tank foam debris caused the Columbia disaster. That’s quite a shift from days earlier when the foam was the “leading candidate” — an explanation that quickly became embarrassing.

Until 1997, Columbia’s external fuel tanks were insulated with a Freon-based foam. Freon is a chlorofluorocarbon (CFC) supposedly linked with ozone depletion and phased out of widespread use under the international treaty known as the Montreal Protocol.

Despite that the Freon-based foam worked well and that an exemption from the CFC phase-out could have been obtained, NASA succumbed to political correctness. The agency substituted an allegedly more eco-friendly foam for the Freon-based foam.

PC-foam was an immediate problem.

The first mission with PC-foam resulted in 11 times more damaged thermal tiles on Columbia than the previous mission with the Freon-based foam.

[Yardstick]

Yeah, I imagine fibrous asbestos embedded in the foam would have reinforced it, kind of like rebar in concrete.

[tanknetter]

I thought of Eileen and her crew and how we had probably just killed them.

Not accurate at all.

One of the post-Columbia procedure changes was that the ISS would be used as a lifeboat in the event of damage that would preclude an orbiter returning safely to Earth, and that ALL shuttle missions were only going to the ISS (after much debate, and over the strenuous objections of the previous NASA Administrator, there was a waiver issued for the last Hubble servicing mission). The ISS was kept stocked up to capacity with consumables, and the Russians stood ready to launch extra Soyuz missions to bring the crew home. Nothing with the foam-shedding on the first post-Columbia return to flight would have impacted the ability of Discover to get to and dock with the ISS.

[SunkenCiv]

Thanks Yo-Yo.

[lacrew]

Same here...the foam, the tiles...makes one wonder how it could possibly go ‘right’.

Forget thermal expansion - the mere weight of the fuel in the tank would cause strain which could crack an inelastic foam, essentially glued to the tank.

[GraceG]

We shouldn’t have built the space shuttle, instead we should have instead concentrated on stamping out Saturn V’s at a “dime a dozen”....

We could have built the current space station in the mid 70’s with only a few launches. We could have had a moon base for what was blown on the space shuttle for crying out loud...

[Vince Ferrer]

We shouldn’t have built the space shuttle, instead we should have instead concentrated on stamping out Saturn V’s at a “dime a dozen”....

The decision to go with the space shuttle was a pragmatic political decision that interest was waning in space flight (it was) and they had to be cost conscious. It didn't turn out that way, but it was an attempt to save money.

[Patton@Bastogne]

.

2012-04-19



"ET" means "Space Shuttle External Tank"
"MMC" means the "Martin-Marietta Corporation
"SOFI" means "Sprayed On Foam Insulation"



I was a SOFI engineer for the ET who warned about low-tensile strength adhesion between the ET's aluminum skin and the foam.

I began work on the ET SOFI Program (Working for a Mike Jonnasson at the MMC Michoud New Orleans facility) in roughly 1984 ...



Standard Quality Assurance (QA) Test Procedures required about one-hundred "plug-pulls" on a completed External Tank (ET).

A "Plug Pull" was basically a one-inch diameter "foam core" that was pulled from the External Tank, to measure the tensile adhesive strength of the SOFI (foam) and the External tank's aluminum skin.

As I recall, the "passing requirement" was about 35 psi ... a value which had presumably been derived from calculations about how strong the foam's adhesion force had to be to properly "stick and hold" to the ET, especially during the Space Shuttle launch sequence.



These one-hundred or so "plug pulls" were always taken from the same standard "foam test locations" on the External Tank.

These areas included:

* Forward Ogive (LO2 tank at the top of the External Tank

* Intertank Section (Structual Cylinder at the Center of the External Tank

* LH2 (Hydrogen Tank) at the Lower Section of the External Tank

* Aft-Ogive Section (Hydrogen Tank) at the very bottom of the External Tank



The test data for the appromimately 30 ETs that MMC had already built (before I started working as a SOFI Engineer) was "suprisingly" stored on paper datasheets (stamped-off by NASA) in cardboard boxes in a dusty records archive at the MMC Michoud Facility in (eastern) New Orleans, where the ET was built.

I had just started studying "Statistical Process Control" (SPC) data analysis (X and R bar caharts, etc), and I also was familiar with computer programming using FORTRAN.



The "paper data sheets" that MMC and NASA had archived were basically useless in the state that I had discovered them.

So I embarked upon a two-stage course of action.

1) place data from the "paper sheets" into a (mainframe) computer database, into data files that could be read by the FORTRAN computer language

2) write an "Expert System" in FORTRAN that would read SOFI test data, from essentially "all" the External Tanks ever built, and then analyize that data, to wit:

2-A) statistical values (average, max, min, standard deviations) of all SOFI foam "plug-pull" tests from all ETs built

2-B) predict specific areas of the ET where the SOFI manufacturing process was deficient (ie. the foam didn't stick well to the ET)

2-C) provide all of this "analyzed data" to MMC and NASA Quality Assurance so that (potential problems could be fixed)



It took about six months to do all of this ... while I continued my daily work as a Manufacturing Engineer in the "Advanced Manufacturing Technology" (AMT) department, managed by Terry Hibbard, with my immediate supervisor a great (honest) man, Mike Johnasson.

Mike Johnasson allowed me to do this special project, presumably knowing where it might lead ...

Looking back, I really admire Mike for that ...



Finally, my data is collected and analyzed, and it appears to point to about six (6) areas on the External Tank that (after 30-plus tanks built) ... have a general tendency for "weak" SOFI tensile adhesion to the ET's aluminum skin ... less than 35-psi required by NASA specifications ...

I present this data to Mike Johnasson, who then (later) takes me to AMT's Terry Hibbard to look at the conclusions.

However, it's decided that there's "no great story" here ... and Management apprears to want to close the issue out ...

I protest, of course (still being a young 33 years old) ... warning explictly that SOFI debris from the ET could damage the Orbiter's Thermal Tiles during the launch sequence, and then possibly have the Orbiter destroyed during re-entry.

The gentle ridicule and cynicism from MMC was difficult to deal with with ...

However, we had just lost the "Challenger" only two years ago ... so I fought like a bulldog ...



Suddenly, one-afternoon ... perhaps 3-4 months later ... I was summoned one afternoon to Ron Heiter's Executive Office (MMC Director), where the following people (as I recall) were present:

1) MMC Legal Counsel

2) Terry Hibbard and Mike Johnasson

3) Ron Heiter, MMC Director (sitting at his large desk).



Ron Heiter told me that a NASA aerodynamics engineer in Huntsville (Marshall Space Flight Center) had analyzed damage to the Space Suttle Orbiter's Thermal-Tiles ... for a large number of launches ...

That analysis had pointed towards SOFI debris (weakly adhered to the ET's aluminum skin) falling-off during the Shuttle launch sequence ...

and then that SOFI debris crashing into the Orbiter's Thermal-Tiles at high-speed ...

damaging, and in some cases, cracking and even knocking the Thermal-Tiles from the Space Shuttle Orbiter.



To everyone's suprise, the analysis used air-flow areodynamic methods to calculate where that SOFI debris was coming from on the External Tank ...

and the areas that he predicted ... corresponded "exactly" to the six-odd areas of "weak foam" that I had predicted (completely independently).



Ron Heiter thanked me for my efforts, and presented me with a $ 500 gift certificate to Campo's Furniture, which I subsequenntly used to purchase a brand-new washing machine and dryer for family.



Of course, the issue remained: "How Do We Fix This" ?

I was (not suprisingly) soon transferred AWAY from the SOFI Engineering area ...

and then terminated (fired, as Mitt Romney likes to say) ...



I took my case directly to NASA's offices, both in New Orleans, and then in Hunstville, Alabama.

No one even bothered to talk to me.



That's my story.

Respecfully.

Douglas A. Coggeshall, Aerospace Mechanical Engineer

CEAI@Engineering-Excellence.US




.

[Patton@Bastogne]

.

2012-04-19



"ET" means "Space Shuttle External Tank"
"MMC" means the "Martin-Marietta Corporation
"SOFI" means "Sprayed On Foam Insulation"



I was a SOFI engineer for the ET who warned about low-tensile strength adhesion between the ET's aluminum skin and the foam.

I began work on the ET SOFI Program (Working for a Mike Jonnasson at the MMC Michoud New Orleans facility) in roughly 1984 ...



Standard Quality Assurance (QA) Test Procedures required about one-hundred "plug-pulls" on a completed External Tank (ET).

A "Plug Pull" was basically a one-inch diameter "foam core" that was pulled from the External Tank, to measure the tensile adhesive strength of the SOFI (foam) and the External tank's aluminum skin.

As I recall, the "passing requirement" was about 35 psi ... a value which had presumably been derived from calculations about how strong the foam's adhesion force had to be to properly "stick and hold" to the ET, especially during the Space Shuttle launch sequence.



These one-hundred or so "plug pulls" were always taken from the same standard "foam test locations" on the External Tank.

These areas included:

* Forward Ogive (LO2 tank at the top of the External Tank

* Intertank Section (Structual Cylinder at the Center of the External Tank

* LH2 (Hydrogen Tank) at the Lower Section of the External Tank

* Aft-Ogive Section (Hydrogen Tank) at the very bottom of the External Tank



The test data for the appromimately 30 ETs that MMC had already built (before I started working as a SOFI Engineer) was "suprisingly" stored on paper datasheets (stamped-off by NASA) in cardboard boxes in a dusty records archive at the MMC Michoud Facility in (eastern) New Orleans, where the ET was built.

I had just started studying "Statistical Process Control" (SPC) data analysis (X and R bar caharts, etc), and I also was familiar with computer programming using FORTRAN.



The "paper data sheets" that MMC and NASA had archived were basically useless in the state that I had discovered them.

So I embarked upon a two-stage course of action.

1) place data from the "paper sheets" into a (mainframe) computer database, into data files that could be read by the FORTRAN computer language

2) write an "Expert System" in FORTRAN that would read SOFI test data, from essentially "all" the External Tanks ever built, and then analyize that data, to wit:

2-A) statistical values (average, max, min, standard deviations) of all SOFI foam "plug-pull" tests from all ETs built

2-B) predict specific areas of the ET where the SOFI manufacturing process was deficient (ie. the foam didn't stick well to the ET)

2-C) provide all of this "analyzed data" to MMC and NASA Quality Assurance so that (potential problems could be fixed)



It took about six months to do all of this ... while I continued my daily work as a Manufacturing Engineer in the "Advanced Manufacturing Technology" (AMT) department, managed by Terry Hibbard, with my immediate supervisor a great (honest) man, Mike Johnasson.

Mike Johnasson allowed me to do this special project, presumably knowing where it might lead ...

Looking back, I really admire Mike for that ...



Finally, my data is collected and analyzed, and it appears to point to about six (6) areas on the External Tank that (after 30-plus tanks built) ... have a general tendency for "weak" SOFI tensile adhesion to the ET's aluminum skin ... less than 35-psi required by NASA specifications ...

I present this data to Mike Johnasson, who then (later) takes me to AMT's Terry Hibbard to look at the conclusions.

However, it's decided that there's "no great story" here ... and Management apprears to want to close the issue out ...

I protest, of course (still being a young 33 years old) ... warning explictly that SOFI debris from the ET could damage the Orbiter's Thermal Tiles during the launch sequence, and then possibly have the Orbiter destroyed during re-entry.

The gentle ridicule and cynicism from MMC was difficult to deal with with ...

However, we had just lost the "Challenger" only two years ago ... so I fought like a bulldog ...



Suddenly, one-afternoon ... perhaps 3-4 months later ... I was summoned one afternoon to Ron Heiter's Executive Office (MMC Director), where the following people (as I recall) were present:

1) MMC Legal Counsel

2) Terry Hibbard and Mike Johnasson

3) Ron Heiter, MMC Director (sitting at his large desk).



Ron Heiter told me that a NASA aerodynamics engineer in Huntsville (Marshall Space Flight Center) had analyzed damage to the Space Suttle Orbiter's Thermal-Tiles ... for a large number of launches ...

That analysis had pointed towards SOFI debris (weakly adhered to the ET's aluminum skin) falling-off during the Shuttle launch sequence ...

and then that SOFI debris crashing into the Orbiter's Thermal-Tiles at high-speed ...

damaging, and in some cases, cracking and even knocking the Thermal-Tiles from the Space Shuttle Orbiter.



To everyone's suprise, the analysis used air-flow areodynamic methods to calculate where that SOFI debris was coming from on the External Tank ...

and the areas that he predicted ... corresponded "exactly" to the six-odd areas of "weak foam" that I had predicted (completely independently).



Ron Heiter thanked me for my efforts, and presented me with a $ 500 gift certificate to Campo's Furniture, which I subsequenntly used to purchase a brand-new washing machine and dryer for family.



Of course, the issue remained: "How Do We Fix This" ?

I was (not suprisingly) soon transferred AWAY from the SOFI Engineering area ...

and then terminated (fired, as Mitt Romney likes to say) ...



I took my case directly to NASA's offices, both in New Orleans, and then in Hunstville, Alabama.

No one even bothered to talk to me.



That's my story.

Respecfully.

Douglas A. Coggeshall, Aerospace Mechanical Engineer

CEAI@Engineering-Excellence.US




.

[Yo-Yo]

Wow. Challenger didn’t have to happen if NASA listened to a lone engineer complaining about O-ring erosion, and Columbia didn’t have to happen if only NASA listened to you (and themselves.)

[tophat9000]

Dont discount the green foam...the shuttle started flying in 81..how many years with no foam loss..then did not change there process they change the foam formula and they started having problem...he just gives the mechanism that impacted the new but not older foam