Mark ở vũ trụ khácVIMark học AI
Mark ở vũ trụ khác · Another Universe Anatomy of a Disaster · Space · Working across teams

Mars Climate Orbiter

57 km lbf·s ≠ N·s

Launched 11 December 1998 · signal lost 23 September 1999 · at Mars orbit insertion

What happenedthe probe dipped too deep into the Martian atmosphere and was never heard from again
Lossone Mars weather satellite; nobody was hurt
Root causea ground software file used pounds instead of newtons
Main sourceMCO Mishap Investigation Board report, November 1999
Reread in 1 minute

Essence

A factor of 4.45One pound-force is 4.45 newtons. The file held pounds; the receiving team read newtons.
The bug was on the groundThe flight software was metric and correct. The error sat in a file made by ground computers.
Nine months of driftA little wrong each time. By arrival the probe was nearly 170 km lower than planned.
Someone noticedNavigators saw mismatched numbers from the spring, but raised them by e-mail, never as a formal incident.
Not one person's slipThe board listed 8 contributing causes, from training to verification.

Source: MCO Mishap Investigation Board, Phase I report (10 Nov 1999), pp. 6–7, 13–17.

Part 1

The question of the case

The interface spec said metric. Navigators saw numbers that did not add up for months. So how did a unit error slip through the handoff between two teams, all the way to the Martian atmosphere?

Dec 1998 – Apr 1999The bad file is bornGround software writes its output in pounds. For four months the file is not even usable.
→
Apr 1999 – Sep 1999Numbers drift, word of mouthDifferent orbit solutions disagree. The worry travels only by e-mail.
→
23 Sep 199957 kmAimed at 226 km, the probe actually dipped to about 57 km. Signal lost.

Source: MCO Mishap Investigation Board, Phase I report, pp. 13–18.

Part 2

What happened

The milestones below come from the Phase I report of the investigation board (chaired by Arthur Stephenson, NASA).

Closest-approach altitude at Mars orbit insertion. Target after the fourth trajectory correction: 226 km. Estimates in the final week: 150 to 170 km. About an hour before the engine burn: as low as 110 km. Estimate after the loss: 57 km. Minimum survivable altitude: 80 km. Mars surface · 0 km 50100150200250 km 80 km · survival limit target (TCM-4) 226 final week 150–170 1 hour before ~110 after the loss ~57
Every recalculation put the probe lower. The board took 80 km as the lowest altitude the probe could survive. The final estimate was 57 km: the probe "either was destroyed in the atmosphere or re-entered heliocentric space".Source: MCO Mishap Investigation Board, Phase I report (Nov 1999), pp. 7, 13–14.
11 Dec 1998 · LaunchA 629 kg probe whose job was to "orbit Mars as the first interplanetary weather satellite" and relay data for the Mars Polar Lander.
Dec 1998 – Apr 1999 · The data file is unusableFor the first four months, the file logging small thruster firings had format problems, so navigators did not use it. From April 1999 it could be used, but the units were still wrong.
Spring – summer 1999 · Numbers disagreeAt the working level there were concerns about orbit solutions that did not match. The Doppler-only solutions "consistently indicated" a path closer to the planet. These discrepancies "were not resolved".
15 Sep 1999 · Fourth trajectory correction (TCM-4)Aimed at a 226 km closest approach.
23 Sep 1999, about an hour before · Estimate of 110 kmA fifth correction (TCM-5) was discussed verbally, but it had never been prepared, so it was not carried out.
23 Sep 1999, 09:00:46 UTC · Orbit insertion burnAt 09:04:52 the probe passed behind Mars, 49 seconds earlier than predicted. The signal never came back.
29 Sep 1999 · The factor of 4.45 is foundSix days after the loss, the small-forces data were found to be too low by a factor of 4.45.

Source: MCO Mishap Investigation Board, Phase I report, pp. 6, 13–14, 17–20; mass of 629 kg: NASA/JPL MCO arrival press kit (Sep 1999), p. 6.

Episode 1

What happened

226 km57 km9 months49 seconds
Episode 2

Why

pounds and newtonsthe handoffe-mail instead of processmyths
Episode 3

Work lessons

check the handoffa mismatch is an incidentplan B
Part 3

Why: four layers of cause

Technical layer: a number crosses the handoff

interface · unitsMCO board, pp. 14–17
The path of the number. The probe fires small thrusters to dump reaction-wheel momentum. Ground software SM_FORCES computes the impulse of each firing and writes it in pound-force seconds. The AMD file is handed over. The interface specification requires newton seconds. Navigators read the numbers as newton seconds, so each firing is underestimated 4.45 times. The orbit estimate drifts. 1 · ground software SM_FORCES computes each small firing output: lbf·s 2 · AMD file handed over interface spec (SIS) requires: N·s nobody checked 3 · navigation team reads the number as N·s feeds the orbit model force 4.45× too small 1 pound-force (lbf) = 4.45 newtons (N) The flight software computed in metric and was correct. The error lived only in the ground model.
The error sat exactly where two teams handed over. The board named the root cause as "the failure to use metric units in the coding of a ground software file, 'Small Forces,' used in trajectory models". The interface spec set the units, but it "was not followed".Source: MCO Mishap Investigation Board, Phase I report, pp. 6, 14, 16–17.

Physical layer: why a small error grew large

angular momentum desaturationMCO board, pp. 6–7, 17, 22
MCO had a solar array on one side of its body, unlike Mars Global Surveyor. So it had to fire small thrusters to dump wheel momentum 10 to 14 times more often than the navigation team expected. Each firing was undercounted, and the error added up over nine months. spacecraft bodysolar array on one side small thrusters dump wheel momentum firings compared with expectation expected: 1 share actual: 10–14 times more often a little short each time · added up over 9 months
New design, old expectations. Compared with Mars Global Surveyor, MCO's solar array was strongly asymmetric. So these firings "occurred 10-14 times more often than was expected by the operations navigation team", and a small error each time became "small errors being introduced in the trajectory estimate over the course of the 9-month journey".Source: MCO Mishap Investigation Board, Phase I report, pp. 6–7, 17, 22.

Process layer: four safety nets, all torn

checks and balancesMCO board, pp. 13, 18–25, 30
Four layers of checks that should have caught the error. One: verification of ground software, unclear whether done independently. Two: the interface specification, not followed and not checked. Three: the Incident, Surprise, Anomaly reporting process, misunderstood by many, with e-mail used instead. Four: the backup trajectory correction TCM-5, never prepared. The error passed through all four. verificationof ground software"not clear" interface specrequired N·s"was not followed" ISA incident formmismatch was seenraised by e-mail only plan Bcorrection TCM-5never prepared The unit error passed through all four layers
Anyone can make an error. Someone has to hold the net. The board wrote that "the team relied on e-mail to solve problems, instead of formal problem resolution processes", so things ended up "slipping through the cracks". It also "observed no fault tree or other a priori analyses of what could go wrong".Source: MCO Mishap Investigation Board, Phase I report, pp. 13 (informal reporting), 16 (spec), 20 (TCM-5), 22–23 (e-mail), 25 (ISA, verification), 30 (fault tree).

Organisational layer: two spacecraft for the price of one

faster, better, cheaperMPIAT, March 2000
Development cost in 1999 dollars. Mars Pathfinder, one landing mission: 196 million dollars. Mars 98, the MCO orbiter plus the Mars Polar Lander: 188 million dollars. The independent assessment team judged Mars 98 underfunded by at least 30 percent. The navigation team ran three missions at once; MCO had a single navigator. Mars Pathfinder (1 mission) 196 Mars '98 (2 craft) 188 million USD development, 1999 dollars "underfunded by at least 30 percent" navigators ran 3 missions at once · MCO had a single navigator
"Faster, better, cheaper" was not wrong, but it was done wrong. Thomas Young's independent assessment team called the slogan "an effective concept … if properly applied", while failing to apply it well "has contributed to an unacceptably high failure rate". The MCO board's second report said the project management team "appeared more focused on meeting mission cost and schedule objectives".Source: Mars Program Independent Assessment Team (MPIAT, 14 Mar 2000), slides 8, 12, 24, 25; MCO board, Phase I, p. 24 (3 missions, single navigator); MCO board, Report on Project Management in NASA (13 Mar 2000), p. 18.

New people, new craft

The navigation team was "unfamiliar with spacecraft" and had "inadequate training" (2 of the 8 contributing causes).

A botched handover

Systems engineering "did not adequately address" the move from development to operations. A recurring question: "Who's in charge?"

Nobody owned the seam

Navigators were worried but "did not communicate their trajectory concerns effectively" to the spacecraft team.

Source: MCO Mishap Investigation Board, Phase I report, pp. 7, 22; Report on Project Management in NASA (Mar 2000), p. 18.

Part 4

Fact check: what is true and what is myth

12 things often said about Mars Climate Orbiter, checked against the primary record. Click each row to see why.

SOLIDA pounds-versus-newtons error in ground software was the root cause

The board's finding (Phase I, pp. 6, 16): the "Small Forces" file did not use metric units and was low by a factor of 4.45.

SOLIDThe probe flew far lower than planned

Aimed at 226 km, final estimate about 57 km, below the 80 km survival limit (pp. 13–14).

SOLIDThe project was short of people and money

The navigation team ran 3 missions at once and MCO had a single navigator (board, p. 24). MPIAT judged Mars '98 "underfunded by at least 30 percent".

CAUTION"Lockheed used imperial, NASA used metric"

Broadly right: NASA's 30 Sep 1999 release says a team in Colorado (industrial partner Lockheed Martin) and a team in California (JPL) used different units. But it was not "NASA metric, the company imperial": the shared spec required metric, and the board does not pin the blame on one company.

CAUTIONThe probe "crashed into" Mars

Nobody knows for sure. The board wrote that it "either was destroyed in the atmosphere or re-entered heliocentric space" (p. 7).

CAUTION"$125 million lost"

This figure comes from 1999 press reports (Reuters, 1 Oct 1999); it is not in the investigation reports. NASA's official numbers cover the orbiter and lander together: $193.1M development and $42.8M operations (NASA/JPL press kit, Sep 1999).

CAUTION"Faster, better, cheaper" was the culprit

MPIAT said the slogan works "if properly applied". The culprit was applying it badly: fewer people, less money, fewer checks.

CAUTIONThe TCM-5 correction would certainly have saved it

Not doing it is one of the 8 contributing causes, and it was designed to raise the orbit to a safe altitude. But it had never been prepared, and the board does not claim it would surely have worked (pp. 19–20).

FALSE"Just one person forgot to convert units"

The board listed 8 contributing causes. NASA's Edward Weiler: "The problem here was not the error, it was the failure of NASA's systems engineering, and the checks and balances in our processes to detect the error."

FALSEThe flight software used the wrong units

"The AMD software installed on the spacecraft used metric units for the computation and was correct." "Mismodeling only occurred in the ground software" (p. 17).

FALSENobody noticed anything until the last minute

From spring 1999 there were working-level concerns about orbit solutions that did not match. They were "only informally reported" and "were not resolved" (p. 13).

FALSEMars Polar Lander was lost to the same unit error

The lander was lost on 3 Dec 1999 for a different reason: the "probable cause" was premature shutdown of the descent engines after a spurious touchdown signal at leg deployment (JPL Special Review Board, Mar 2000).

Sources: MCO Mishap Investigation Board, Phase I (Nov 1999); NASA release 99-113 (30 Sep 1999); MPIAT (Mar 2000); NASA/JPL press kit (Sep 1999); Reuters (1 Oct 1999); JPL D-18709 (22 Mar 2000).

Part 5

Assessing the case

Scores are Mark's assessment based on the sources in Part 8, not figures from a study.

Avoidable
5/5
Warnings in advance
4/5
Open investigation
5/5
Lessons retained
3/5
Done right afterwards
  • Preliminary cause made public just 7 days later
  • Said plainly: the system failed, not one person
  • Recommendations applied at once to Mars Polar Lander
Not done right
  • 10 weeks later Mars Polar Lander was lost too (different cause)
  • The "forgot to convert" story spread faster than the process lesson
  • Fixing the incident-reporting culture was slower than fixing a data file

A very ordinary error that had to pass through four layers of checks to become a disaster. That makes it far more worth studying than a joke about units.

More: two spacecraft, two failures, one root

Mars Polar Lander was lost on 3 December 1999. JPL's review board concluded the software probably mistook a spurious signal at leg deployment for touchdown, allowing engine shutdown at about 40 m altitude. A completely different bug, but the independent MPIAT team put both losses in the same picture: Mars '98 was short of money, short of people, and light on checks.

The MCO board's second report (March 2000) set out the vision "Mission Success First" and called for an environment where "problems may be raised without fear of reprisal".

Sources: NASA release 99-113 (30 Sep 1999); JPL D-18709 (22 Mar 2000); MPIAT (14 Mar 2000); Report on Project Management in NASA (13 Mar 2000), pp. 6, 37.

Part 6

Work lessons: check the handoff between two teams

Applies wherever one team passes numbers, files or an API to another: finance and sales, backend and frontend, supplier and customer.

Four steps to check a handoff: put the unit in the data itself, test end to end with real data, treat a mismatch as an incident and log it, and prepare plan B before you need it. 1 · unit in the data"force_Ns", not "force"a machine can check it 2 · test end to endone real case, both sidescompare the results 3 · mismatch = incidenta ticket with an ownernot an e-mail thread 4 · prepare plan Bwhen to use it, who decideswritten before you need it Having a spec is not enough. Someone has to check that it is followed.
Each step patches one torn net from MCO. Units were not checked, ground software had no end-to-end test, the mismatch went by e-mail, and TCM-5 had never been prepared.Source: the nets follow the MCO Mishap Investigation Board, Phase I report, pp. 16–25; the steps are Mark's suggestions.
Every seam has one ownerNot team A, not team B, but one person accountable for both ends.
Don't resolve incidents by e-mailE-mail drifts away. A ticket has a status, an owner and a deadline.
Builders stay on into operationsThe board recommends key people "remain with the project from the start to operations".
Ask "what if it's wrong?" firstDraw a fault tree before launch. The board found no such analysis for MCO.

Source: Report on Project Management in NASA (Mar 2000), p. 40; MCO board, Phase I, pp. 22–23, 30.

People sometimes make errors. The problem here was not the error, it was the failure of NASA's systems engineering, and the checks and balances in our processes to detect the error.Edward Weiler, NASA Associate Administrator for Space Science, release 99-113 (30 Sep 1999).
Part 7

Further reading

MCO Mishap Investigation Board, Phase I Report (10 Nov 1999)Section 5 on the contributing causes is gripping. Better than any retelling.
MCO Mishap Investigation Board, Report on Project Management in NASA (13 Mar 2000)Project management lessons drawn from the case, usable in any field.
Thomas Young's MPIAT report (14 Mar 2000)The whole Mars programme: money, people, and the "faster, better, cheaper" slogan.
Anatomy of a Disaster #03: The Tenerife collision (1977)Two Boeing 747s on one fog-bound runway, and words that were not quite clear enough.
Part 8

Sources

Primary documents first. Page numbers follow the PDF and may be off by one. Where only a press source exists, it is flagged in the Fact check section.

  1. Mars Climate Orbiter Mishap Investigation Board, Phase I Report, 10 Nov 1999. llis.nasa.gov
  2. Mars Climate Orbiter Mishap Investigation Board, Report on Project Management in NASA, 13 Mar 2000. discovery.larc.nasa.gov
  3. NASA, release 99-113, "Mars Climate Orbiter team finds likely cause of loss", 30 Sep 1999. nssdc.gsfc.nasa.gov
  4. Mars Program Independent Assessment Team (chair T. Young), report, 14 Mar 2000. PDF copy
  5. NASA/JPL, Mars Climate Orbiter Arrival press kit, Sep 1999. jpl.nasa.gov
  6. JPL Special Review Board, Report on the Loss of the Mars Polar Lander and Deep Space 2 Missions, JPL D-18709, 22 Mar 2000. PDF copy
  7. NASA NSSDCA, Mars Climate Orbiter entry (1998-073A). nssdc.gsfc.nasa.gov
  8. Reuters, "NASA Says Human Error Caused Loss Of Mars Craft", 1 Oct 1999 (press source for the $125 million figure). archived copy