Apollo 13 questions

[Willoughby]

It is my understanding that all the service modules burned up on reentry since they were jettisoned after being on course to reenter.  This would of course include the Apollo 13 service module.  There are fairly detailed explanations on what happened inside of the oxygen tank that exploded.  I understand that a lot of this would simply come from telemetry data such as voltage, heat, etc., but the actual physical defect in the venting system that is described - is this just an assumption based on the data - because it couldn't be from physical examination. 

Also, did mission control have the ability to remote fire the SPS engine?  I wonder if there was ever a discussion of firing it after jettison to see if it was actually damaged - whether it would work or not or if it still would have been too dangerous in close proximity to the recently jettisoned command module.

Since they didn't fire the engine, do we know with any degree of certainty whether or not the SPS engine would have worked - or is that still as much a mystery now as it was when they made the decision not to risk firing it up?  I ask this because I come across several people who make the claim that had the Apollo 8 astronauts experienced a similar event - obviously without the benefit of the LM's descent engine - then they would have surely died.  I've made the argument that this isn't necessarily the case, that they just would not have had the option to avoid the risk.  Of course the engine could have been damaged and exploded - or not fired at all which would have resulted in their deaths, but couldn't it have possibly worked and returned the astronauts safely?  EDIT :  They still would have run out of oxygen and power in the CSM, so nevermind the last part...

[JayUtah]

I understand that a lot of this would simply come from telemetry data such as voltage, heat, etc., but the actual physical defect in the venting system that is described - is this just an assumption based on the data - because it couldn't be from physical examination.

They're inferences from telemetry, but pretty accurate and reliable inferences.  The operation of an Apollo spacecraft was predicated on being able to infer likely causes and modes of failure based on telemetry, and instrumenting the spacecraft accordingly.  If you plan to operate something based on on-site or remote inspections you have to plan for that in the design phase, and that would have been hard to do for much Apollo hardware.  So while it was an unanticipated failure, the telemetry coupled with knowledge of the spacecraft design was the intended means for diagnosing failure.

However, the SM was photographed from the CM after separation.  The resolution is not super, owing to the wide-angle lens and the distance.  But a knowledgeable person can draw further conclusions from the disposition of the wreckage as seen in the photos.

Also, did mission control have the ability to remote fire the SPS engine?

None whatsoever, after CM separation.  And a fairly limited ability otherwise, in the normal configuration.  For unmanned test flights, the software and hardware were configured to allow for it.  In normal flight, the SPS would need to be armed (a physical switch in the cockpit) and the computer would need to be set to accept ground commands.

...it still would have been too dangerous in close proximity to the recently jettisoned command module.

A very good point.  The priority at that stage of the mission was the safe return of the astronauts.  Engineers were lucky to get photographs.

Since they didn't fire the engine, do we know with any degree of certainty whether or not the SPS engine would have worked - or is that still as much a mystery now as it was when they made the decision not to risk firing it up?

While the crew visually noted damage to the SPS nozzle, it remains unknown whether the SPS engine would have safely operated.

I ask this because I come across several people who make the claim that had the Apollo 8 astronauts experienced a similar event - obviously without the benefit of the LM's descent engine - then they would have surely died.

A number of informed people have noted that had the accident occurred at most other points in the Apollo 13 mission, the crew would likely have perished.

EDIT :  They still would have run out of oxygen and power in the CSM, so nevermind the last part...

Right: the problem was consumables, not so much propulsion.  A course correction had to be made, of course, to transfer from the hybrid trajectory to the free-return trajectory.  The second major maneuver was to accelerate their return, but that too was a decision made on the basis of consumables budget.  It could have been omitted.  But the FRT correction was essential.  Without it, they would have missed Earth and gone into solar orbit.  So without the LM propulsion, they would have had little choice but to attempt the maneuver with the SPS.

[smartcooky]

A number of informed people have noted that had the accident occurred at most other points in the Apollo 13 mission, the crew would likely have perished.

Two periods of time seem logical to me

1. Any time prior to LM extraction and docking would likely have been terminal because there would be no way to extract and dock quickly enough for the crew to board the LM.

2. Any time after LM separation in Lunar Orbit would also be terminal, because there would again, not be enough time to re-dock the LM and use it as a lifeboat

I have a question though. I have read various articles that describe the wiring fault in No. 2 Oxygen Tank as making it effectively a bomb waiting to go off. Stirring the tanks was the trigger on A13, but could it had gone off at any time, were there other actions that the crew might have done which would trigger it, or was the stirring of the tanks the only possible trigger?

[JayUtah]

I have a question though. I have read various articles that describe the wiring fault in No. 2 Oxygen Tank as making it effectively a bomb waiting to go off. Stirring the tanks was the trigger on A13, but could it had gone off at any time, were there other actions that the crew might have done which would trigger it, or was the stirring of the tanks the only possible trigger?

There was no wiring fault.  There was no single fault, really.  There isn't even a single root cause.  This is why I teach this over and over to my junior engineers.

The precipitating event was the botched unshipping of the tank.  They dropped it and damaged the purge system.  This led to them trying a previously untested detanking procedure later using the tank heater on ground power.  Ground power had been upped to 60 VDC from the standard 28 VDC used in most aeronautical systems.  The belief was that the thermostat would work, but the testing to verify the thermostat at 60 VDC was incomplete.  It was validated at 60 VDC with the contacts closed.  Arcing was not part of the test plan, because contact separation was driven by thermal parameters, which did not change as a result of the modification to ground power.  The thermostat tripped correctly at 80 F, and promptly arced and fused shut.  That led to thermal runaway in the heater, which burned off key parts of the wiring insulation.

Once you have uninsulated wires, any energization of them has the potential to cause an arc in the presence of LOX.  Stiring, sure.  Heater activation, sure.  Even the sensor transducers could have done it.  It's all a matter of what newly-uninsulated wires are close enough to each other to allow for an arc.

[smartcooky]

There was no wiring fault.  There was no single fault, really.  There isn't even a single root cause.  This is why I teach this over and over to my junior engineers.

The precipitating event was the botched unshipping of the tank.  They dropped it and damaged the purge system.  This led to them trying a previously untested detanking procedure later using the tank heater on ground power.  Ground power had been upped to 60 VDC from the standard 28 VDC used in most aeronautical systems.  The belief was that the thermostat would work, but the testing to verify the thermostat at 60 VDC was incomplete.  It was validated at 60 VDC with the contacts closed.  Arcing was not part of the test plan, because contact separation was driven by thermal parameters, which did not change as a result of the modification to ground power.  The thermostat tripped correctly at 80 F, and promptly arced and fused shut.  That led to thermal runaway in the heater, which burned off key parts of the wiring insulation.

OK. Not so much a wiring fault as wiring damage caused by the use of incorrect procedures.

Once you have uninsulated wires, any energization of them has the potential to cause an arc in the presence of LOX.  Stiring, sure.  Heater activation, sure.  Even the sensor transducers could have done it.  It's all a matter of what newly-uninsulated wires are close enough to each other to allow for an arc.

So this could have gone boom at any time. Jack Swigert was just unlucky enough to be the guy who flicked the switch.

[bknight]

I have a question though. I have read various articles that describe the wiring fault in No. 2 Oxygen Tank as making it effectively a bomb waiting to go off. Stirring the tanks was the trigger on A13, but could it had gone off at any time, were there other actions that the crew might have done which would trigger it, or was the stirring of the tanks the only possible trigger?

There was no wiring fault.  There was no single fault, really.  There isn't even a single root cause.  This is why I teach this over and over to my junior engineers.

The precipitating event was the botched unshipping of the tank.  They dropped it and damaged the purge system.  This led to them trying a previously untested detanking procedure later using the tank heater on ground power.  Ground power had been upped to 60 VDC from the standard 28 VDC used in most aeronautical systems.  The belief was that the thermostat would work, but the testing to verify the thermostat at 60 VDC was incomplete.  It was validated at 60 VDC with the contacts closed.  Arcing was not part of the test plan, because contact separation was driven by thermal parameters, which did not change as a result of the modification to ground power.  The thermostat tripped correctly at 80 F, and promptly arced and fused shut.  That led to thermal runaway in the heater, which burned off key parts of the wiring insulation.

Once you have uninsulated wires, any energization of them has the potential to cause an arc in the presence of LOX.  Stiring, sure.  Heater activation, sure.  Even the sensor transducers could have done it.  It's all a matter of what newly-uninsulated wires are close enough to each other to allow for an arc.

Cascading errors, but what I would ask at this point:
If the ground power was configured to 28 vdc would there have been any likely errors to an including the venting of O2 through the bent valve?

[nikolai]

Jack Swigert was just unlucky enough to be the guy who flicked the switch.

Or lucky enough, as he did it at a time when the resultant problem was survivable.

[Obviousman]

If the ground power was configured to 28 vdc would there have been any likely errors to an including the venting of O2 through the bent valve?

I believe the procedure would have worked as expected, and Apollo 13 could have been just another moon landing.

[JayUtah]

OK. Not so much a wiring fault as wiring damage caused by the use of incorrect procedures.

Maybe not even incorrect procedures.  The procedure for removing the cryogenic oxygen tank was correct and well documented.  But it was not correctly carried out:  a bolt was left in place that, according to procedure, should have been removed.  That's a human error, not an error in the procedure.  The forklift operator who was to perform the final lift and extraction could have realized sooner that the tank was binding.  But it wasn't supposed to be binding at all, so who was at fault?  Would a more skilled operator have done better?

Once the tank had dropped two inches back onto the shelf, the correct procedure was to send it back to Beech Aircraft for inspection and requalification.  Beech's inspection revealed a damaged purge assembly.  But the purge assembly is not a flight-critical element.  It's used only on the ground during detanking, which never happens in flight.  The people who made those decisions did not consider the consequences of ad hoc procedures that might be employed later by others to compensate for the degraded purge assembly.  To a certain extent, they are allowed to assume that all correct procedures will be followed from there on out.

The 60-volt qualification test was also done at this time.  But that is an electrical test only.  The engineers determined that 60 volts DC could be applied to the various components of the tank without any operation that exceeded the flight limits, including the thermostat (in the closed position).  The thermal trip test is a different test.  The thermostat doesn't have to be energized for that test, and it was not required for the 60 VDC qualification anyway.  In hindsight we can certainly argue it should have been, so there's an example of an incorrect procedure.

A normal tank vent/purge cycle does not require the heater.  You simply connect the ground purge assembly and open the purge valve.  A technician monitors tank pressure on the ground service equipment (GSE) panel to verify when the tank has completed venting.  Then you purge with something like gaseous nitrogen or ambient air through the fill valve for a prescribed amount of time to eliminate the possibility of an oxygen concentration in the tank.  The tank is then considered "safed."  The cycle has to complete before other operations near the SM can happen, such as those that might produce sparks.  So there's some pressure (no pun intended) for it to happen according to schedule.

The decision to turn on the tank heater to speed up boil-off was approved by the relevant engineers, so by (pedantic) definition it was not an incorrect procedure.  A significant part of my day on any given day is approving variances to established procedure.  As soon as a procedure deviation bears my signature, it has just as much validity with respect to regulation as an approved standard procedure.  The engineers who signed off on this procedure had every reason to believe that it was safe to do so because the Beech engineers had qualified the tank for 60 VDC GSE operation, and the procedure did not contemplate operating the tank outside the recommended limits.  Sensibly enough, the tank temperature gauge on the GSE did not read any higher than the tank was designed to go, even when heated.  Not incorrect procedure, but maybe short-sighted.

The concept at play here is something called "tolerance buildup."  We approve decisions in engineering based on tolerances for error, safety factors, and so forth.  A tolerance buildup occurs when individual components drift toward an out-of-tolerance condition in all the same direction, so that the result is an entire system that is dangerously close to an overall tolerance violation.  At every step of the process in servicing and operating the cryogenic tank, a little step was taken toward making the tank a little less tolerant of heat and electrical conditions.  Individually, most of them were innocuous.  The only truly out-of-tolerance condition that was allowed was the runaway heater during the detanking.  That resulted in a truly intolerable condition:  uninsulated wires.  But up until then, there was just a sequence of, "Yeah, I suppose that's okay."

So this could have gone boom at any time. Jack Swigert was just unlucky enough to be the guy who flicked the switch.

Yup.

[JayUtah]

If the ground power was configured to 28 vdc would there have been any likely errors to an including the venting of O2 through the bent valve?

No.  The thermostat would have functioned as expected and prevented the heater from baking the tank.

Ironically the plethora of voltages used in the spacecraft and launch vehicle and the likelihood of human error in operating GSE connections was what motivated standardizing GSE DC voltage to 60 and validating the entire spacecraft to accept that voltage for direct DC connections.  It's not clear from my reading that the GSE could even have been configured to supply a lesser voltage.

[bknight]

If the ground power was configured to 28 vdc would there have been any likely errors to an including the venting of O2 through the bent valve?

No.  The thermostat would have functioned as expected and prevented the heater from baking the tank.

Ironically the plethora of voltages used in the spacecraft and launch vehicle and the likelihood of human error in operating GSE connections was what motivated standardizing GSE DC voltage to 60 and validating the entire spacecraft to accept that voltage for direct DC connections.  It's not clear from my reading that the GSE could even have been configured to supply a lesser voltage.

Were the changes substantial to the spacecraft?

[JayUtah]

Were the changes substantial to the spacecraft?

Not really, because as I go back to the original documents I discover my memory isn't correct.  The GSE was still able to supply a variety of voltages, both AC and DC, including the nominal 28 VDC that the tank's original design assumed.  So had they used 28 volts instead of 65 (not 60) volts to accelerate the detanking, they would have been fine.  Not all spacecraft systems were upgraded to 65 volts.  The possibility that a misconfigured GSE power supply could silently damage the spacecraft remained a serious concern, but the problem was addressed by changes both to the spacecraft systems and to the GSE, whichever was more straightforward in each case.