William Gerstenmaier, “Transitioning from the Space Shuttle to Constellation” - MA Space Grant Consortium Public Lecture at MIT (4/15/2009)

[MUSIC PLAYING]

HOFFMAN: My name is Jeff Hoffman. I am the director of the Massachusetts Space Grant Consortium. Space Grant is a NASA-sponsored organization involving over 750 universities nationwide of which we have 18 here in Massachusetts who are members together with the Museum of Science and the Christa McAuliffe Challenger Education Center.

Space Grant is interested in aerospace education, in increasing the workforce of trained aerospace workers, engineers, especially systems engineers. It is also a presence that NASA has in the many states which don't actually have NASA centers.

And one of the things which the Massachusetts Space Grant has been doing, really, over the last 20 years-- if you look at the last page of the handout, you'll see a list of very distinguished lecturers who have come here to talk about their work in aerospace. And this year, it's our great pleasure to welcome William Gerstenmaier who is the NASA associate administrator for space operations.

I'm not going to give you a complete biography. It's printed out on the inside. Probably all of you have had a chance to read it. But I just want to remind you, first of all, that with the responsibility for spaceflight operations, that means that Bill is responsible for the operation of a space shuttle and the International Space Station.

So you can do the arithmetic yourself of how many billion dollars a year he's actually responsible for and how many billions worth of accumulated hardware he is responsible for. It is a very serious responsibility. And I think NASA and the country is really very fortunate that in Bill they've found somebody who is not just a competent-- more than a competent-- excellent administrator, but is also a first-class engineer.

I actually first got to know Bill way back in the early 1980s when he was a propulsion engineer at the Johnson Space Center. And he admitted when we were talking about that earlier today that that's still his deep love. And he is still a working engineer. And he is very much interested in the engineering aspects of all these very complex systems that he is responsible for.

In fact, the title of the talk that we use as a working title for a long time is "The Transition From the Space Shuttle to the Constellation System." And he'll have a few words to say about that at the end.

And we'll certainly answer a lot of questions about it, but, in fact, one of the critical aspects of the transition from the space shuttle to the constellation system is the continued safe operation of the space shuttle itself, which is no easy task. You're all aware that the space shuttle is extremely capable, but it's very complex. It's getting old. And over the years, we've had a lot of problems, which have had to be addressed.

And as an example of the sort of continuing engineering problem that has to be dealt with in the safe operation of the shuttle, Bill is going to talk about a recent systems engineering problem which had to be solved. So I think this is a really nice talk here at MIT that you're going to hear, really, about a systems engineering problem of operational importance for the space shuttle. So with that, I'm going to plug you in, Bill, and invite you to come up for your talk.

GERSTENMAIER: Thank you. All right. So, again, thanks, Jeff. And thanks to everybody that took time to come here. I'll keep this kind of informal.

And I'll talk a little bit about a systems engineering problem, as Jeff talked about. And I'll answer any questions on transition and other things, but I think it's good that you get a chance to maybe hear kind of behind the scenes how we solve problems with the space shuttle, and how we fly in space, and what things are there.

So I'll take this example. This is the flow control valve issue. I jokingly call this my senior design project. I'll start out with kind of a chronological order of how this unfolded. And I won't do it in hindsight. I'm going to do it as it occurred to us in the engineering community and the shuttle and station programs on a day-to-day basis.

So I could go to the end. And then you would get the Reader's Digest version-- and we fixed the valves, we flew them, and everything was okay. But that's not the way it really happened.

So now you get a chance to see how it unfolded. And then you'll get a chance to see how we use system engineering tools, how we use the team, how we used expertise, how I used folks in Mississippi, people in Cleveland, people out in California, all together as a coherent team to go solve this problem.

So with that, you ought to get an understanding of kind of where we are overall. And I'll answer any kind of questions at the end.

HOFFMAN: So it's the one closer to you that makes it--

GERSTENMAIER: Yep. All right. Other way? Oops. Other way. Here we go.

HOFFMAN: Yeah, because here's--

GERSTENMAIER: Here we go. Okay.

HOFFMAN: That's the laser if you need it. And the middle one is forward.

GERSTENMAIER: Okay. So this is a schematic of the shuttle. This is that big orange tank you see out on the outside of the shuttle in simplified schematic. It has an oxygen tank and a big hydrogen tank.

The system that we had the problem in is in the hydrogen system. And the way that the shuttle works is we have the three main engines. And each one of the main engines, we tap off the hydrogen that cooled the outer jacket here. We tap that off, and we run it through a little thing called a flow control valve.

And this valve, this tiny little valve, controls all the pressure that goes into that tank. And I should have brought that-- I had the valve with me in my briefcase, but I forgot it, so we didn't bring the valve.

But it's a little small valve about the size of this device. And there's three of them in there. And they control all the pressurization of that tank, and so they're obviously a very critical component.

So if you take a look at where they live, there's three of them. They live in the aft compartment of the space shuttle. They're in these three locations. They're latch valves 56, 58, and 57. They sit in those three locations in the aft compartment of the shuttle.

So the main engines-- the plumbing goes from there, up to these valves. Then it runs up the entire length of the tank, about 170-foot run of pipe. Goes back down in and drops into the top of the external tank. And their purpose is to keep the tank pressurized and at the proper operating pressure.

So if you take a look at them, this is what they look like in real life. They're about this size, the size of this pointer. They're made up of shims. They sit-- oops-- and this picture kind of shows a little more detail of where they are.

Let me just go back one. They have shims on them. I don't think there's anything else.

Here's a solenoid that keeps them in place. They're here. This is the way that the flow comes down, comes out of the main engines, goes in here, then flows to this little area, then flows out, in and out that 170-foot run to the tank. You'll notice that the entire poppet stroke is 170/1,000 of an inch. So it hardly moves. It also never goes fully closed.

It's always open at some point to keep the tank there, because, if you think about it, the pressurant that's coming in is coming from the main engines that pressurizes the tank that is then feeding the same main engines. So it's a closed loop. So therefore, it never actually has to ever fully close.

And so when this solenoid is activated-- or energized, it pulls this valve in, 170/1,000, slows down that flow, keeps the tank pressurized. And then when it's turned off, it goes to the high-flow state. So with that, you know everything about flow control valves. So you can check that off on your resume somewhere.

Now, this is what we saw in-flight. So what we saw in-flight was this is one valve, operated fine. This is the outlet pressure-- looked okay. This is the center valve, latch valve 57.

It went to essentially high flow and stayed at high flow until we got main engine shut off. This third valve, it shut down when this one went to high flow, because all three of them tee together, and they flow into the tank.

So when this one failed to open, this one compensated by dropping down. Everything was fine, absolutely no change, performed exactly the way it should. Totally nominal engine performance. Everything looked good. The system did exactly what it should do.

We looked at this in-flight after the first eight minutes. We saw it right away. We assumed, ah-ha, the electrical system broke. The solenoid didn't work. The valve went open. We'll fix the wires. We'll go fly again the next day. No big deal.

And you will note that that was November 15, 2008. And again, not in hindsight-- this is as it occurred. So we looked at this. No big deal. Easy. Must be a loose connector, loose wire. The valve didn't fully close, so life was okay.

Now, December 19, we go into the aft compartment. We X-ray into that all that plumbing you saw there. We stick a couple of X-ray plates underneath there. We put a radiation source on the outside. We shine the X-rays through there, and we get this picture.

Now, if you look at this picture, to the trained eye, you can't really tell what you're looking at. But instead of seeing a full, round circle here, you can see there's, like, a piece missing. So if you look over here on this picture, there's a full, round, like, quarter-sized piece. And there's this chunk missing.

And what we see in the X-rays, we saw this chunk missing. So that's what gave us a clue is, hey, there's something wrong here. There's a piece missing to this valve. It looked like it failed. We pulled it out, and this is what we found.

So just a little tiny piece. And if I had it with me-- it's really small. It broke off. And then that allowed it, essentially, to mimic high flow.

And then the other valves shut down. So the failure was this little piece broke off, and now our problem was-- from December 19-- is to figure out what caused that little piece to break off.

So the first thing we did is we said well, we'll go look upstream. See if there's anything upstream that could have flowed down into the valve. So we looked upstream, and lo and behold, there's this big chunk of braise here.

And if you look here, this is the outside of the tube where it's braised in place. There was a hot spot when it got braised. So the person that braised it braised it and overheated it. It put some anomalous gold blob on the inside.

It looked like a piece that separated off of here. We call that the separation gap. It looked like a piece of this had broken off. So our hypothesis on December 23rd, four days later, was that a piece of this material broke off, went down, hit the valve, broke the valve, and therefore the piece came off.

So that was the theory on December 23rd. We'll find out that this was a red herring and didn't have anything to do with the problem. But it was an interesting thought at the time.

So then we further looked upstream. And again, what this is is this is the flow control valve. This is the flow in. Then it goes here. It hits this 90 degree corner which is essentially a block of metal. Then it goes around another 90 degree corner. Then all three of them T in this manifold. And then they go up that 170 foot line up to the external tank.

So we took a borescope and went in there and we looked. And lo and behold, here's two little ding marks right where the particle hit. So it came in, hit this corner, bounced off, hit this corner here, and then went up here. It made no marks on this area. Made no marks on any of this tubing. But yet, down in this area, it took this little chunk of metal out down in this area.

Now the reason this is important was there is a concern that first of all, if this piece breaks off, can it overpressurize a tank? That would be one failure. The other concern was that it turns out that the flow out of this piece-- and I'll show you some CFD plots in a minute-- but the flow out of here is about 1,000 feet per second.

So even though this particle weighs just a couple grams, it's just like a rifle bullet going down this tube. And it didn't do much damage. But could it actually damage the tube here? And that's a question we're going to have to answer later.

Could it then come up here, damage a tube? Could it ricochet here? It's still traveling 700 feet per second here at this point. It's 700 or so feet per second even out here. And then it's about 600 feet per second all the way up that 170-foot line into the tank.

So that kind of velocity could potentially penetrate this tubing. So now we've got to be careful. If another one of these breaks off, could it then rupture a piece of tubing and cause a hydrogen leak? Which wouldn't be a good deal.

So then the next thing we did is we started taking a look at our valves to see if we could see anything. So we go to a scanning electron microscope and we take a look at our valves. And this is what we see on the surface.

And when the shuttle was designed, we really didn't have the capability to do scanning electron microscope inspections like we do today. So we look at this. And this is pretty much that machine surface.

So if you look at where that valve comes together, it's just like the quarter pieces up here. And then it comes together in this little radius where that little quarter piece sits. So this is a 2000 times x-ray in that little radius.

So what you're seeing here is just basically the machining marks of how that little poppet is machined around there. So we didn't see anything here that we thought was too anomalous or too big a deal. But you can tell that there's a lot of machining marks in here that could mask some things if you do the typical kind of inspections that we do.

And at this time, all we used to do is we would put dye on the material and we would shine a fluorescent scope on there and we would look for cracks or imperfections. It's not a very sophisticated technique, but it's good to find little cracks and little imperfections. And this was on January 14th when we did this test.

So then we started looking at it from a structural dynamics standpoint. And we started looking at the flow. And as I described to you, the flow across here is about 1,000 feet per second. It goes through a very small orifice.

And it turns out that potentially, that flow can couple into this lip. And this thing can vibrate at a high cycle, maybe 100,000 Hertz, and it could actually break off.

So we had this theory at this point that maybe the failure was associated with this extremely resonant condition that could occur periodically. We had not changed these valves from the beginning of the shuttle program back in 1981. So intuitively, we didn't think it could be a problem.

But maybe there's some coupling that can occur that this thing can couple into that natural frequency and just break off a piece. It would be essentially an uncontrolled breakup as that occurs.

So this was the first kind of analysis we had back on January 21st that started making us get a little bit concerned that maybe there's something going on here more than we had really thought.

So then we went back and we looked at our failure history, and our failure history was kind of sordid. We didn't have many failures. We had two failures that occurred at acceptance test programs.

So these valves had not flown anywhere. They just went into the test rig. We were doing acceptance tests to make sure that they were ready, and they failed. And the crack size was-- whoops-- the crack size was 57 degrees, or that little arc was a 57 degree arc.

And in this case, it was a 72 degree arc. And then the failure that we saw in flight, this was an 87 degree arc and it had flown 11 times. And then this one, we went back in our inventory and we looked, and we could see a crack.

And it was a fairly large crack in there. About 270,000th, 0.27. Almost a quarter of an inch crack that was very visible even with our dive penetrate activities. We had pulled this valve out for an anomaly back on STS-97. We hadn't looked at the valve. We pulled it out now and looked at it and lo and behold, there was a crack there.

So we had this data, and this is all the data we had. And what we were struggling with was, were these initial failures here with 0 flights-- was that this resonant thing that I just showed you on a previous page or was it something else going on with these valves?

Or were these outliers? Were these manufacturing defects that we essentially screened out with this acceptance test procedure? And we didn't really know at this time. So what we're trying to do here is we're trying to look back at past history and relate to what we to see if we can tease out what the root cause is, or what the root cause of the failure is.

So again, now what we're trying to do is we're trying to data mine. So it was unique to this valve engine position. So we plot the three engine positions. We plot where these valves have flown. We're essentially data mining to see if this natural frequency shows up in only one engine location.

Does it show up in others? Is it a function of the inlet/outlet diameters? It a function of the valve housing? We're just looking at raw data to try to understand if we can find any correlations. And we spent a ton of time with this and we couldn't find anything.

We then kind of kept looking a little bit. We pulled back the acceptance test procedure failures. And these occurred back in 1990. And if you look at them a little bit in these pictures, you could see that there's initiation site, and then they pretty much just break in a high-cycle fatigue kind of thumbnail. This is very much what a crack looks like in a thumbnail section.

This one over here, this is the one from flight. And you can see in it, it evidences a couple of different zones. You see a zone 2, zone-- excuse me. A zone 1, a zone 2, and then a zone 3. And the zone 3 is the actual failure.

So when we look at this, it's another view of the same thing. You can see the various thumbnails going. This is very typical of a high-cycle fatigue. So this isn't a one time kind of failure. This doesn't occur immediately.

This is telltale evidence that it occurred over a large number of cycles. So this appears to us to be some kind of high-cycle fatigue issue, but not an immediate failure. But our ATP cases-- those all clearly look like they just immediately failed in the first application of loads. You put a load on it and it failed.

So we use this data to kind of say that well, maybe these are just outliers. We'll throw that data out. We've got some kind of high-cycle fatigue problem. We're probably okay to fly as long as we screen for cracks is kind of what we thought at this time.

Now I will tell you that is what I thought. I have a large team that supports me. And part of my team carries the title safety, so they have to think the opposite way I do no matter what I say.

And they're thinking, it doesn't matter. This is just nice pictures. You're crazy. It's the same failure. It can happen anytime. We're grounding the fleet. So then we're having wonderful discussions about all this. But that's life. And that's actually good.

So then we said, okay. Now if they're going to tell me it can happen on any flight, I'll go look at the consequent side. So what I did now is I went down to-- a team went down to New Orleans down to the Space Flight Center, and we essentially mocked up the 79 feet of tubing. So we actually put 79 feet of tubing exactly the same size there.

We put an orifice in a line. We put a couple of camera ports in. We put some laser measurement ports in so we can get the velocity right. And we built a little system to drop particles into this system. And we were firing particles at 700 feet per second down this tube.

And so we fired about 200 or 300 particles down this tube. And you can see here, this is what it looked like in the worst bend, which was this first elbow here. Just little scuff marks. No big dings. No big hardware.

And you can see here if you look really hard, we actually bowed the metal out a little bit with one of the shots. So it's a kind of a probabilistic game of if they fly exactly the right way and they hit flat edge on, they're just going to scrape. But if they go knife edge on, they can really cause some damage. So even though this is pretty compelling test data, it still wasn't good enough necessarily for my analysts.

So then at this time we said, okay. Can we do a better job of inspecting? So what you see up here is-- whoops. What you see up here is we figured out a way to take that little edge where that little metal is and we actually polish it with diamond dust.

So where you saw all those little machine marks before, if we put diamond dust on a little small piece of cloth and we put a toothpick on there, we can actually shine that piece up. So then we do the scanning electron microscope, it's not hidden by the machining marks and we can look for cracks. So this was a brand new technique that we kind of developed on the fly to go do this diamond polishing to not destroy it.

Now the other problem I had was I couldn't remove a significant amount of metal here or I would invalidate my flight hardware. Because the shuttle program's ending, I took away the flow-balancing rig that allows me to set the solenoid strength with the flow through the valve to the proper current so it moves in the right direction.

So I didn't have a way, once I removed metal from these, to ever fly these valves again. So once I actually even did this little bit of polishing, the valve was no longer certified for flight. So I couldn't go look at all my flight valves with this technique or I would invalidate it and it wouldn't be a piece of flight hardware.

Now I'm in the process now, and the flow rig is up now and it's available today. But we would've missed that launch window. We would've been after the Soviets. We would probably be trying to launch about this time now with the shuttle mission if we would've destroyed our flight assets.

So I'm now constrained. So this is a different problem than you'd like to have as an engineer. I would like to go take my best inspection technique and use it, but I can't because if I use that best inspection technique, I've lost the ability to go fly that piece of hardware. So that's another consideration that we face in the engineering world that sometimes you don't face in the outside world.

So then we took a look at our older valves, which we could easily do. These are the-- or excuse me, these are the newer valves. And because I didn't have the new inspection technique, I'm looking at these with dye penetrate and I'm seeing no cracks.

There's no cracks in any of my new valves that I'm flying. The only crack I've ever had was this one that actually broke on flight on STS-126. So now I'm starting to build rationale that maybe there's something between my old valves and my new valves that are there.

But I'm really blinded here because I have a very bad inspection technique. I just had this dye pen inspection technique. I don't have a way to really look for the very, very small cracks.

And so then what we're doing here is that now we can actually start to see some of these cracks in these older valves. And this gives you a feel for how small we're looking for. The big back line up here is 2/1000ths of an inch.

This crack width down here is about thousandth of an inch . And these are really small. And so if you think about it, I do this diamond dust on the lathe. I polish it up. And then I take 4,000 images around the top of the valve.

And then I have someone manually inspect all 4,000 of those images looking for cracks on the order of 1/1000th of an inch to maybe 1/10,000th of an inch in length.

And so at first I go to my guys, why is it taking you, like, two days to inspect my valves? You should be able to get these pictures and get them back. And then they showed me the 4,000 images and they asked me if I would like to go through those 4,000 images in the next half hour and give them an answer. And I said okay, I understand.

So then we went back again. And this is what we saw with this technique in the older valves. We saw lots of cracks. I mean, they were everywhere. Now the other thing that's important is these valves have flown since the first shuttle flight, so there has been no change from the first shuttle flight.

So we are flying with the same risk whether we realized it or not since STS-1 back in 1981 until today. And because we didn't have a good inspection technique, we thought everything was okay.

But in reality, here's what we were flying with. And this is the one that we found later. This is a big crack. A quarter of an inch long. We even saw that with a dye pen. The other ones we couldn't see.

But we had a significant amount of cracks in our valves, which means potentially we could be losing a lot of these if they couple into this flow. So there's definitely something on here we need to be looking at. This is a serious problem. We've got to really understand this before we can go back flying.

So again, this is before polishing. So you can see all the machine marks here. Then down here, you can see it after polishing. And then you can see, this is what the crack looks like. And then I blew it up over here. So again, you can see the crack over here on this side.

So this is a very good technique. This really gives us a great way to go look at these valves. The problem is that it invalidates the valve as a flight valve. So this is the technique I really want to use but I can't use. So again, just another picture of the crack again.

Now we're again back looking at some of these. So when we found these cracks, we've now got the ability to now load the edge of the valve. So now I put it in an Instron machine which you guys are familiar with, and I tack on a little strut on the end. And now I'm going to pull on the valve until it fails.

And so what I'm doing is I'm pulling on the valve with 250 pounds, 12,000 cycles, and I get a failure like this. I pull on the valve 175 pounds for 3,000 cycles, I get failures like this. And so what I'm doing here is I'm trying to understand, is this high-cycle fatigue where we get the three zones of fracture? Is that real?

And you can start to see in these pictures, it shows the various failure zones just like we saw before. So this looks like a very promising failure mode. So this I'm going to try to use with my engineering team to say, I'm ready to go fly.

What I'm going to try to tell them is if I dye penned, I didn't see any cracks. I didn't have-- I've got maybe 12 flights on these valves. Nowhere near these number of cycles. I'm a long way away from a failure. That's good enough to go fly. Let's go fly.

So this is the rationale I'm about ready to go try with my team. And you'll see at the end the timeline of where I failed with this rationale. But at least I tried.

So again, this is just the same thing. You can see this is the part that actually failed. The three zones we talked about-- the four zones. And then down here, I did another overload. And here what I did is to get it to fail, I notched a little piece. So I notched the two corners, and then I overloaded it and it popped.

And why this is important is the crack ran, and it would have ran further except it ran it into the notch that we put in to make it fail. So the problem here was I had to also limit the size of the failure.

Because if I lost a big enough piece, the particle testing where we're shooting particles down the tube showed that I could penetrate a piece of tube and cause a hydrogen leak which we thought at the time was a big deal. So I was again trapped.

And I had to bound the size. I had to show that enough life in these parts would [INAUDIBLE] and I could go fly. And that's the rough flight rationale we were trying to get together.

And then my wonderful dynamicists, the CFD guys, came along. And they did this plot. And what they found here was that as the engines throttle up and down, the valve position moves a little bit. And it turns out that depending on where we are in the throttling case and the material, you can either couple into one of these dynamic modes or not couple in this dynamic mode.

So going back to the original theory where I showed you that you could get the single event that fails, they're now giving me a population why it doesn't occur very often. It only occurs when I'm in certain throttling conditions, and I have to stay in that throttling condition long enough for it to fail.

So then we went and we said, well, okay. We'll try to fire some bigger particles. So we fired them into that elbow. And you can see right here, I shot a hole through the elbow, which wasn't such a good thing.

I actually got so good at sending these things down, I could actually get them to stick in the elbow, which was not a good thing. But my test team was exceptionally proud of this. They calculated ahead of time what velocity they needed to get it to stick into the elbow. And then they beat me $5 that it would stick in the elbow. And it stuck in the elbow and I paid them $5.

[LAUGHTER]

But I didn't feel so good about that. So we had good analysis, but it showed that, hey. This is really a potential problem. And you can see the same thing here. We also put holes. And this is what the elbow looks like in itself. We manufactured a bunch of these. And in Cleveland, we actually shot these particles at them to see what would happen.

So then finally, this is kind of coming to the end, is that what happened was we had these assets. And then luckily, my team kept working on another inspection technique, and I'll show you some pictures of it in a minute. It's eddy current.

And they were able to use eddy current to define cracks. So these are my old valves. These are the ones where we saw the cracks before. We had previously not seen any cracks here. But now I can get eddy current indications that show that there are cracks even in my flight valves.

So I got a new inspection technique that allowed me to now determine that there were very small cracks present in some of these valves. So now I had a method to go hand-select valves to show that there were no cracks with any detectable method of inspection. And that's eventually how we got to go to flight.

Again, this just shows the various NDE techniques. Polishing and etching, mag particle, eddy current, and then scanning electron microscope. They're kind of all different techniques, but probably the scanning electron microscope is the definitive one if you can look at the images. Eddy current turned out to be very, very good, and I'll show you a picture of that in a minute.

But again, I didn't have that technique until the very end-- until the March timeframe, just before flight. And what happened was I didn't even know that this existed. I was about ready to throw in the towel. I told my team we're not going to fly unless we can inspect.

So therefore, we're going to have to fix the flow-balancing rig. We're going stand down. We're not going to fly for a couple months. And they go, well, we've got this new technique we've been working on. We haven't told you about it. It's called eddy current.

We'd like to try it on these valves. If we can and it works, would you accept that as flight rationale? And I said, well, okay. So then they went ahead and did that.

And what they did is they went out and they bought a bolt tester. And they had done this at Christmas, so maybe this is a Christmas gift. They bought this bolt tester, and it has a very nice little probe on it that fits right-- this is the poppet-- it fits right into the area of problem.

And then the valve scans around. And in about 20 minutes, they get a trace of the entire interface. And it looks for imperfections in there by looking at the eddy current changes and it can detect the failures. And it turned out to be very reliable.

And this is an actual screen sample of the picture. And even I can find the cracks that are here. And then they repeat down here again because they actually rotate the part around twice to confirm them.

But this is very good. It no damage to the part. And it's 100%-- it's actually better in some ways than the scanning electron microscope because it doesn't need the human to go find It. So with this tool, I had the way to go select valves. And this was the way we ultimately ended up going to flight.

At the same time, when I punched a hole in the corner-- we never give up at NASA, right? So we had designed this. And this is a doubler. So we built this little device that we could install in the orbiter out at the pad and actually go fly with. It carries a little O-ring in here, so when I punched a hole in the corner it seals itself and will not leak.

So I had these built. They're certified. They're ready to go fly. We chose not to fly, but they're available if we want to go fly them. So if you can't fix the problem itself, can you then put some kind of splint or Band-Aid on the outside that makes the consequences less for you? And that was the intent of this device, but we ended up not having to use that.

So then basically where we ended up with what we think is going on is hydrogen-assisted, high-cycle fatigue. So the fact that it's in a hydrogen environment, it's embrittling that interface. So when a small crack starts, it hardens in that interface, and then cycles, and then hardens a little more and cycles and hardens a little more, and it finally relieves itself.

So we're pretty confident that we can fly valves without cracks, we have a good flight rationale. And that's going to be our rationale for probably the remaining number of flights on the shuttle.

We do PRA. So we looked at the probability of the poppet breaking, the probability of the size being big enough to cause damage, the probability of an impact, and the probability of actually causing some kind of damage. And in this is loss of crew and vehicle. So we're looking at probably with the wear-out case, we think we have about a 1 in 450 risk of flying with this design.

So again, I'm going to go through this, but this is basically the chronology of what I just talked to you about. But what you see through here is-- oops. Again, I tried to get the flight rationale here. We do a big flight readiness review.

There was no way I was going to get the resolution here. We waited until-- the next chart-- we waited till here on February 20th. I thought I had good enough rationale there. That didn't work.

And then we finally got eddy current here, and then that was the thing that actually allowed us to go fly down here or at least attempt to fly on March 11th. And then we ended up scrubbing on that flight for an unrelated hydrogen leak.

But the thing you ought to take out of this in terms of messages-- you didn't hear about any of this in the press. All you hear about in the press is the shuttle didn't fly. Some aging hardware problem, right?

And it wasn't an aging hardware problem. It was a design problem. The design problem was there from the beginning of the program. We were exposed to the same risk all along.

We had seen this a couple times in other failures but we didn't pursue it to this level. And now that we finally pursued it to this level, I think we understand this problem and we know where we are.

Now the story that's not up here is we flew successfully. We got back, we eddy currented the valves. They were fine. They were crack-free. I then got my flow bouncing facility up. I put one of the valves in the flow balancing facility. And after the flow bouncing facility, a crack showed up.

So this implies to me that there's still this loading phenomena going on. The flow-balancing facility uses nitrogen, it uses higher pressures and higher flow rates, so it can definitely stress these valves. So we're still going to use the same rationale to scan no crack-free and then go fly. We're also looking at alternate material to go change.

So hopefully through all this, you kind of got an idea of what it takes to actually get a shuttle to launch and what my life is as a program manager or AA at headquarters. Now as AA, I'm not supposed to do any of this stuff, as Jeff said.

I'm not supposed to understand anything technically. I'm supposed to be a politician that goes and talks to all the White House staffers and the Executive Office of the President and all those folks.

But that isn't my passion. My passion is this stuff. They've momentarily got me stuck in DC. So rather than talking about shuttle phase out, transition, politics, and those kind of things, I'd rather talk about engineering.

So hopefully out of all this, you learned something about how we do systems engineering. This is a huge effort. The team is probably about 1,000 people supporting this. They're in Cleveland. They're in Louisiana. They were in California. They were in Marshall Space Flight Center.

I've got fluid dynamics guys. I got materials guys. I got NDE guys working on this, and gals. I mean, it's an amazing team of pulling all this effort together to go essentially from this problem we saw really in December until flight in March, which is a pretty phenomenal amount of work in three months.

I probably fired over maybe 1,000 particles at tubes. Computer Cray runs on the order of probably a couple hours out at Ames, looking at the CFD stuff. So the amount of effort and work that went into this is phenomenal. The amount of weekends, the amount of holiday time that the team's put in is just unbelievable.

So you don't see any of that. But then when this flies, I can tell you, I was not looking out the window in Florida at the shuttle launch. I was looking at the data of the flow control valves and watching those pressures at the outlets to see how the valves were doing.

And I didn't really care about looking out the window. I knew what I needed to go look at in terms of data. So again, that engineer tendency comes through.

So with that, I'll conclude and open it up to any kind of questions you have. Anything you want to talk about just feel free to ask, and I'll talk to you about any subject that you'd like. So thanks.

[APPLAUSE]

Yep.

AUDIENCE: Are you going to expand this validation method of using the bolt tester to test other components on the shuttle you might have similar worries about?

GERSTENMAIER: Yep. We sure are. It turns out-- we looked at the oxygen valves. I didn't show you, but there's a very similar system on the oxygen side that pressurizes the oxygen system. We also tap the oxygen off and feed it in the oxygen tank. We're doing exactly the same thing with those valves. And the technique works really well.

So this bolt tester was actually designed by a company I think in the oilfield industry to look at bolts-- to look at bolt heads. So this tool is designed very nice. It has two little, tiny probes that sit right on the edge. It makes it just perfect for doing this task for this design.

So that's another huge lesson learned, is you don't need to go out and invent your new technique or your new concept. See what other folks are doing in a totally different industry, and can you then adapt it to the problem that you've got?

These guys didn't know, but we gave them I think, $20,000 to go by this bolt tester and it ended up saving a shuttle flight by probably six months of effort.

So we will definitely do that. We're going to stay-- I call it staying hungry. You just need to keep looking at all these things. So when you get a little anomaly or a little problem, that's a gift.

So then you got to take that gift and expand it as far as you can to understand what it's really trying to teach you. And don't just say, that's a one-of-a-kind failure and blow it off, because later I guarantee you it ll come back.

Because typically when a major failure occurs, you've seen indications of it coming along the way and you've ignored those for a variety of reasons. And I think you guys see that probably in your own testing in your own labs.

Yeah.

AUDIENCE: Bill, how concerned were you that the team becomes myopically focused on this one problem and that other problems that need attention in the same reviews don't get the same amount of attention?

GERSTENMAIER: Extremely concerned. And so they get lectures from me. It's called kids soccer lecture. And in kids soccer, wherever the ball goes, the little pack of kids moves to the ball, right?

So I tell them that that's not how you win soccer games. You win soccer games by being strategically placed across the field. And the ball may be nowhere near you, but you need to be there so when the ball gets there, you're ready.

So it's the same thing here. okay, you guys focus on the flow control valve. This other broader team, you better be positioned across that soccer field and be ready for that next problem that's coming.

Because I get very concerned that you get this tunnel vision that you're working so hard on this one problem, that's all we talk about. And we don't worry about all the other 100,000 things on this shuttle that could be a problem, too. So that's a very good point. Yep.

AUDIENCE: At what point where you convinced that this thing would fly?

GERSTENMAIER: Well, I was convinced probably at the second flight readiness review. So the first flight readiness review, I was not ready yet because there was still enough uncertainty.

But then by the second one, I had enough of an indication that I was relying on flight history, I was using it smartly, that we weren't seeing this all the time. I couldn't believe that we could couple into this every time. And I thought our dye pen penetration measurement was probably good enough to go fly.