What Caused The Explosion That Crippled Apollo 13?

[smartcooky]

Another excellent Scott Manly video.

This one dives into the details of what caused the rupture of oxygen tank no. 2 in the service module of Apollo 13?

[Ranb]

James Lovell's book, Lost Moon, also goes into detail about the cause.  If I recall correctly, Lovell was told about the difficulty with decanting the oxygen tank.  When he was told that replacing and testing another tank for the service module would make them miss their launch window, he was okay with the tank in the condition it was in. 

Of course they thought it was just a damaged drain tube and not damaged internals from the overheating while boiling off the oxygen after the test.

[Obviousman]

The prime cause - IIRC - was the missed change from 28VDC to whatever it changed to. If that had been caught, and the tanks modified, then the decant process would have been OK.

And perhaps that the gauge didn't properly reflect the temperature inside the tank during the process, but if an expected temp range is up to 100C then you really have to think about why you would have a gauge that goes to say, 500C.

Edited to add: but then again, I am not an engineer so perhaps I'm wrong there.

[Kiwi]

Apollo By the Numbers has a section on how the problem occurred near the end of its Apollo 13 Summary.
https://history.nasa.gov/SP-4029/Apollo_13a_Summary.htm

How the Problem Occurred

Following is a list of factors that led to the accident:

* After assembly and acceptance testing, oxygen tank 2, assigned to Apollo 13, was shipped from Beech Aircraft Corporation to North American Rockwell (NR) in apparently satisfactory condition.

* It is now known, however, that the tank contained two protective thermostatic switches on the heater assembly, which were inadequate and would subsequently fail during ground test operations at Kennedy Space Center (KSC).

* In addition, it is probable that the tank contained a loosely fitting fill tube assembly. This assembly was probably displaced during subsequent handling, which included an incident at the prime contractor’s arc plant in which the tank was jarred.

* In itself, the displaced fill tube assembly was not particularly serious, but it led to the use of improvised detanking procedures at KSC which almost certainly set the stage for the accident.

* Although Beech did not encounter any problem in detanking during acceptance tests, it was not possible to detank oxygen tank 2 using normal procedures at KSC. Tests and analyses indicated that this was due to gas leakage through the displaced fill tube assembly.

* The special detanking procedures at KSC subjected the tank to an extended period of heater operation and pressure cycling. These procedures had not been used before, and the tank had not been qualified by test for the conditions experienced. However, the procedures did not violate the specifications that governed the operation of the heaters at KSC.

* In reviewing these procedures before the flight, officials of NASA, NR, and Beech did not recognize the possibility of damage due to overheating. Many of these officials were not aware of the extended heater operation. In any event, adequate thermostatic switches might have been expected to protect the tank.

* A number of factors contributed to the presence of inadequate thermostatic switches in the heater assembly. The original 1962 specifications from NR to Beech Aircraft Corporation for the tank and heater assembly specified the use of 28 V DC power, which was used in the spacecraft. In 1965, NR issued a revised specification which stated that the heaters should use a 65 V DC power supply for tank pressurization; this was the power supply used at KSC to reduce pressurization time. Beech ordered switches for the Block II tanks but did not change the switch specifications to be compatible with 65 V DC.

* The thermostatic switch discrepancy was not detected by NASA, NR, or Beech in their review of documentation, nor did tests identify the incompatibility of the switches with the ground support equipment at KSC, since neither qualification nor acceptance testing required switch cycling under load as should have been done. It was a serious oversight in which all parties shared.

* The thermostatic switches could accommodate the 65 V DC during tank pressurization because they normally remained cool and closed. However, they could not open without damage with 65 V DC power applied. They were never required to do so until the special detanking. During this procedure, as the switches started to open when they reached their upper temperature limit, they were welded permanently closed by the resulting arc and were rendered inoperative as protective thermostats.

* Failure of the thermostatic switches to open could have been detected at KSC if switch operation had been checked by observing heater current readings on the oxygen tank heater control panel. Although it was not recognized at that time, the tank temperature readings indicated that the heaters had reached their temperature limit and switch opening should have been expected.

* As shown by subsequent tests, failure of the thermostatic switches probably permitted the temperature of the heater tube assembly to reach about 1,000° F in spots during the continuous eight-hour period of heater operation. Such heating was shown in tests to severely damage the Teflon insulation on the fan motor wires in the vicinity of the heater assembly. From that time on, including pad occupancy, oxygen tank 2 was in a hazardous condition when filled with oxygen and electrically powered.

* It was not until nearly 56 hours into the mission, however, that the fan motor wiring, possibly moved by the fan stirring, short circuited and ignited its insulation by means of an electric arc. The resulting combustion in the oxygen tank probably overheated and caused a failure in the wiring conduit where it entered the tank, and possibly in a portion of the tank itself.

* The rapid expulsion of high-pressure oxygen which followed, possibly augmented by combustion of insulation in the space surrounding the tank, blew off the outer panel into bay 4 of the SM, caused a leak in the high-pressure system of oxygen tank 1, damaged the high-gain antenna, caused other miscellaneous damage, and aborted the mission.

[JayUtah]

The prime cause - IIRC - was the missed change from 28VDC to whatever it changed to. If that had been caught, and the tanks modified, then the decant process would have been OK.

65 VDC, for ground equipment.  The tank hardware was tested at 65 VDC and flight-qualified based on those tests.  The problem is not that the design wasn't qualified for the higher voltage.  It was that the test regime did not test both critical factors together.  The thermostat could carry 65 VDC safely without modification.  The problem was when it tripped at 80 F.  But the voltage test and the trip test are two separate test rigs, so they aren't tested together.  The trip test uses only a small test voltage to verify that the circuit was broken.  The essence of that test rig was an immersion to simulate actual heat transfer.  It's the thermal and mechanical properties of the switch that you want to test with that rig.  Conversely the voltage test is a simple bench test.  It is possible to test the arcing behavior using that rig, but the test procedure steps didn't call that out.  (Yes, a "TPS Report" is something that actually -- no kidding -- existed at NASA.)

And perhaps that the gauge didn't properly reflect the temperature inside the tank during the process, but if an expected temp range is up to 100C then you really have to think about why you would have a gauge that goes to say, 500C.

But such gauges aren't as accurate in the commonly-occurring range.  If you have a car speedometer that goes to 150 mph, but most of your driving is at 0-10 mph and you need to know how fast you're going to an accuracy and precision of 0.1 mph, then that gauge doesn't help you.  However, a severe-overtemp indicator would have been a good design.  Something that trips and stays tripped when the temperature at any time went above a critical level where you can infer that damage probably occurred.

[smartcooky]

But such gauges aren't as accurate in the commonly-occurring range.  If you have a car speedometer that goes to 150 mph, but most of your driving is at 0-10 mph and you need to know how fast you're going to an accuracy and precision of 0.1 mph, then that gauge doesn't help you.  However, a severe-overtemp indicator would have been a good design.  Something that trips and stays tripped when the temperature at any time went above a critical level where you can infer that damage probably occurred.

Indeed.

The inaccuracy of gauges such as those using a moving coil and spring (e.g. an ammeter or voltmeter) are usually expressed in terms of "percentage of full scale deflection" (%FS). If a gauge goes up to 500 FSD and is accurate to 1%FS, this means a reading of 500 could actually be 495 to 505, but a reading of 50 could be 45 to 55 - an inaccuracy of 10% at THAT deflection.

And that's just the meter end of the system. In the case of a thermal reading, there is also the inaccuracy of the thermal transmitter to be taken into account. Not sure how the Apollo Oxygen tank system worked, but most thermal measurement systems that terminate in a meter usually involve a thermal sensor that that has a voltage output related to the temperature.. mV/°, and there will be an accuracy rating for that as well   

[Peter B]

Murray & Cox's book (Apollo - The Race to the Moon) goes into a fair bit of detail about the underlying causes of the accident.

I suppose one thing which occurs to me is this: on the one hand Joe Shea pushed his engineers to nail down the design for Apollo, knowing they could potentially tinker with/improve the design for years if allowed to; on the other hand there was a rigorous change management process to try to catch unintended consequences of any design alterations.

So was the underlying cause of the Apollo 13 accident an almost inevitable consequence of a design that could have been improved but had to be locked in at some point (even with possible flaws), or a failure of the change management process? Or is it a bit of each?

[JayUtah]

You put artificial limits on design exercises because engineers will inevitably continue tinkering forever if you don't.  it's a consequence of engineering as a commercial pursuit, where schedule and budget become variables in the engineering management equation.  The engineer wants to minimize operational risk by reworking the design.  But it's more accurate to say you want a design that's just good enough, because better is the enemy of good.  Often an engineer will have to defer to someone else's opinion of "good enough."

I think it's more helpful to look at the test regime.  While designs are locked down at a certain point, acceptance and flight qualification test criteria are locked down even earlier -- in the requirements analysis stage.  The contractor and the customer agree on what criteria will need to be met in order for the contractor to get paid, and for the article to be ready for flight.  In most cases the methods of testing will be agreed upon.  Once set, these factors are hard to change.  It's the same philosophy as engineering change management, in the sense that you don't want to enact some ad hoc change to a testing regime that ends up missing something important.  So proposals to change the test regime have to be reviewed for technical validity.  But from the business standpoint, neither party wants the other tinkering with the acceptance criteria.  The customer doesn't want the contractor relaxing the standards, and the contractor doesn't want the customer to move the goalposts.  Because there are both technical and business concerns at stake, changing tests is at least as difficult, if not more so, than changing designs.

I don't know if testing the thermostat trip behavior under load was actually on anyone's radar at the time, but it probably should have been.  And if it was, it may have been working its way through the change management process.  To answer your actual question, I would have to say a little of both.  Whether the work product in question was the tank design or the testing plan, both would have been locked down fairly early in the process.  And any proposed revision to them -- no matter how logical, important, or well-intentioned -- would need to go through a rigorous change-management process.

[bknight]

It seems to me that schedule over rode safety procedures.  The ground crew tested both tanks and only #2 didn't evacuate to ambient pressures indicating that something was wrong with the tank.  The pressure to launch showed up in the two shuttle  disasters.  I was in the oil/gas drilling industry for over 50 years and when something in the operation failed, I always deferred to caution/safety, but then I didn't have Congress breathing down my neck, just impatient bosses. 

[molesworth]

By coincidence, someone on another forum (nothing to do with spaceflight) just posted a link to a real-time "experience" of the mission - https://apolloinrealtime.org/13/

I haven't explored it yet, but it looks interesting.