Mark ở vũ trụ khácVIMark học AI
Mark ở vũ trụ khác · Another Universe Anatomy of a Disaster · Finance · Software

Knight Capital

45 min

Knight Capital Americas · 1 August 2012 · New York Stock Exchange

Whatone server fired orders nonstop for about 45 minutes
Damageabout $460 million; the firm nearly went under
Root causeold code woken up on 1 of 8 servers that missed an update
Main sourceSEC Order No. 34-70694, 16 Oct 2013
Re-read in 1 minute

Essence

1 of 8 serversNew code was copied server by server. One was missed, and nobody checked.
A reused flagThe flag for the new feature used to switch on "Power Peg", code unused since 2003.
Orders that never stop212 customer orders became over 4 million executions, 397 million shares.
97 unread e-mailsA system said "Power Peg disabled" from 8:01, but those messages were not designed as alerts.
No stop buttonNo incident procedure. A hasty rollback broke all 8 servers.

Source: SEC Order No. 34-70694 (16 Oct 2013), paragraphs 13–17, 19, 21 and 27.

Part 1

The question

A firm handling about 10% of trading in U.S. listed stocks, living on software, lost almost half a billion dollars from one update. Why? Not because of one careless technician, but because a deployment process had no safety net at all.

Source: SEC Order 34-70694, paragraph 11 (about 10% of trading) and paragraph 17 (the loss).

2003–2005Dead codePower Peg retired but left on the servers. Changed in 2005 without retesting.
→
27 Jul 2012Unchecked deployNew code copied to 8 servers. One missed. Nobody checked.
→
1 Aug 201245 minutesServer 8 fires orders nonstop. The firm holds billions in positions.
Part 2

What happened

Timeline from the SEC order; the loss from Knight's own releases.

Morning of 1 August 2012. From about 8:01, an internal system sends e-mails reporting "Power Peg disabled", 97 in all before the 9:30 open. At 9:30, server eight starts sending child orders continuously. In about 45 minutes: over 4 million executions in 154 stocks, over 397 million shares. Midway, Knight removes the new code from the seven good servers, so all eight run the old code. End: net long about 3.5 billion dollars and net short about 3.15 billion dollars. 97 "Power Peg disabled" e-mails not treated as alerts about 45 minutes of runaway orders 8:01first e-mail 9:30 · openserver 8 fires orders new code removedfrom 7 good servers→ all 8 broken positionslong $3.5 bnshort $3.15 bn 212 customer orders → over 4 million executions · 154 stocks · over 397 million shares illustrative time axis, not to scale · U.S. Eastern time
The fault spoke up before the open. 97 e-mails named Power Peg outright, but they were not designed as alerts and were generally not read. The SEC gives no exact end time, only "approximately 45 minutes".Source: SEC Order 34-70694, paragraphs 16–17 (45 minutes, 212 orders, 4 million executions, 154 stocks, 397 million shares, positions), 19 (97 e-mails, 8:01) and 27 (rollback).
2003 · Power Peg retired, not deletedThe old feature was no longer used. The code stayed in the SMARS order router and could still be called if the right flag was set.
2005 · Changed without retestingThe part that counted filled shares, which told orders when to stop, was moved. Power Peg was not retested.
Oct 2011 · A different warningTest data was mistakenly used for live quotes, a loss of nearly $7.5 million. Knight fixed that spot and did not look wider.
From 27 Jul 2012 · New code rolled outFor NYSE's new Retail Liquidity Program (RLP). The new code reused Power Peg's flag. A technician did not copy it to 1 of the 8 servers.
1 Aug 2012 · About 45 minutesServer 8 runs the old code and sends child orders nonstop for each parent order, never learning it is filled.
2 Aug 2012 · Knight speaksA "technology issue" "related to Knight's installation of trading software", a pre-tax loss of about $440 million.
6 Aug 2012 · Rescued$400 million of preferred stock from six investors, convertible at $1.50 a share.
16 Oct 2013 · The SEC concludesA $12 million penalty. The SEC's first enforcement action under the market access rule (Rule 15c3-5).

Source: SEC Order 34-70694, paragraphs 13–15 (2003, 2005, rollout), 33 (Oct 2011 incident), 16 (1 Aug); Knight releases of 2 Aug 2012 (8-K) and 6 Aug 2012; SEC press release 2013-222.

Part 1

What happened

45 minutes8 servers97 e-mails$460 million
Part 2

Why

reused flagunchecked deployno stop buttonmyths
Part 3

Work lesson

checklistsdelete dead codekill switch
Part 3

Why: five layers of cause

Technical layer: one flag, two meanings

repurposed flag · dead codeSEC, paras 13–16
A customer parent order arrives with the RLP flag on. On the seven updated servers the flag triggers the new RLP code and the order is handled correctly. On the eighth, un-updated server the same flag triggers the old Power Peg code, which had lost its filled-share counter in 2005, so it sends child orders without end. customer order flag = ON 7 updated servers flag → new RLP code server 8, missed flag → Power Peg (2003) stops when filledhandled correctly never sees fillschild orders forever
The new code was not wrong. It ran correctly on 7 servers. The danger was that the old code was still there, and the new flag was the old flag. Knight intended to delete Power Peg but had not.Source: SEC Order 34-70694, paragraph 13 (reused flag, intent to delete Power Peg), 14 (2003, 2005, not retested) and 16 (7 servers correct, server 8 sending orders continuously).

Deployment layer: one server missed, no second pair of eyes

no peer review · no written procedureSEC, paras 15 and 26
Eight SMARS servers. Seven received the new code. The eighth did not. No second technician reviewed it, no written procedure required a review, and there were no written code development and deployment procedures for SMARS. newnewnewnewnewnewnew OLD missed ✗ no second technician review ✗ no written procedure requiring one ✗ no written code deployment procedures for SMARS (other groups had them)
One person's slip, the whole process's gap. The SEC did not stop at "a technician did not copy the code". It added: nobody was assigned to check, and nothing in writing required a check.Source: SEC Order 34-70694, paragraph 15 (1 of 8 servers, no second technician, no written procedure) and 26 (no code development and deployment procedures for SMARS).

Amplification layer: from 212 orders to billions

amplificationSEC, paras 17–18
212 parent orders became over 4 million executions in 154 stocks, over 397 million shares. Final positions: net long about 3.5 billion dollars in 80 stocks, net short about 3.15 billion dollars in 74 stocks. Loss about 460 million dollars. In 75 stocks prices moved more than 5 percent, in 37 of them more than 10 percent. 212parent orders → > 4 millionexecutions → > 397 millionshares · 154 stocks net long~$3.5 bn · 80 stocks net short~$3.15 bn · 74 stocks loss~$460 million prices: 75 stocks moved over 5% · 37 over 10% (in those 37, Knight was over half the volume)
Automation amplifies mistakes too. The SEC wrote that without proper controls, automation can "turn an otherwise manageable error into an extreme event".Source: SEC Order 34-70694, paragraph 3 (quote), 17 (orders, positions, $460 million loss) and 18 (price moves).

Monitoring layer: signals, but no alarms

signals that are not alarmsSEC, paras 19, 21–25
Three layers of monitoring and none sounded an alarm. 97 error e-mails were not designed as alerts. The main risk tool, PMON, relied entirely on people watching, with no automated alerts. The 33 Account had a 2 million dollar limit not linked to automated blocks. SMARS also did not compare orders going out with orders coming in. 97 BNET e-mails "Power Peg disabled" not an alertgenerally not read PMON main risk monitor human eyes onlyno automated alerts The "33" Account $2 million limit not wired to auto-blockover the limit, still running and SMARS did not compare orders out with orders in.
A system that relies on human eyes is slower than a high-speed order machine. The limits existed on paper, but none was wired to an automatic block.Source: SEC Order 34-70694, paragraph 19 (97 e-mails, "not designed ... to be system alerts"), 21 (no output monitoring), 22–24 (limits, 33 Account) and 25 (PMON).

Response layer: no stop button, and the rollback made it worse

no kill switch · no incident playbookSEC, paras 21 and 27
Before the rollback: one of eight servers runs old code. Knight removes the new code from the seven good servers, intending to go back to the old state. After the rollback: all eight run old code, because the flag in the orders still switches on Power Peg. before rollback 1/8 servers broken remove new code after rollback 8/8 servers broken no procedure to halt SMARS on aberrant activity · no incident-response procedure fixing it live during trading, relying mainly on the technology team
"Roll back" is only safe if the old version is safe. The flag in customer orders was still on, so removing the new code meant waking Power Peg on the other 7 servers. Knight had automatic shutdown for certain strategies of one trading group, but not for SMARS.Source: SEC Order 34-70694, paragraph 21 (no procedure to halt SMARS), 27 (no incident response, rollback made it worse) and footnote 7 (automatic shutdown for certain strategies).

Paper said "compliant"

The initial compliance assessment concluded Knight's controls satisfied Rule 15c3-5 (SEC, paragraph 28).

Old incident, narrow fix

The nearly $7.5 million loss in October 2011 was patched on the spot, with no broad review (SEC, paragraph 33).

Soft signature

The CEO certification (March 2012) said Knight had "processes" to comply, not that its controls complied (SEC, paragraph 34).
Part 4

Fact check: facts and myths

12 things often said about Knight Capital, checked against the SEC order and Knight's own SEC filings. Click a row to see why.

Three numbers for the loss. On 2 August 2012 Knight estimated a pre-tax loss of about 440 million dollars. On 17 October 2012 the third-quarter report showed a trading loss of 457.6 million dollars, 461.1 million including fees. In 2013 the SEC wrote 460 million dollars. 2 Aug 2012 · Knight estimate~$440 million 17 Oct 2012 · Q3 report$457.6 million trading loss 16 Oct 2013 · SEC"more than $460 million"
Three numbers, not a contradiction, just different dates. 440 was the next-day estimate; 457.6 is the booked figure; 460 is the SEC's. The orders poured out in about 45 minutes; the loss took shape as Knight traded out of the positions afterwards.Source: Knight release of 2 Aug 2012 (8-K, Exhibit 99.1); Knight Q3 2012 results release (17 Oct 2012); SEC Order 34-70694, paragraphs 1 and 17.
SOLIDThe faulty orders ran for about 45 minutes

The SEC says "approximately 45 minutes" in paragraphs 1, 10 and 17. It gives no exact end time, so this piece gives none.

SOLIDThe bug came from code unused since 2003 and changed in 2005 without retesting

SEC, paragraph 14: Knight stopped using Power Peg in 2003; in 2005 it moved the filled-share counter and "did not retest" Power Peg.

SOLIDRolling back the new code made it worse

SEC, paragraph 27: Knight removed the RLP code from the 7 good servers; incoming orders then triggered Power Peg on all 7.

SOLIDKnight handled about 10% of U.S. listed stock trading

SEC, paragraph 11, for 2011–2012. SMARS alone was about 1% or more.

SOLIDA $12 million penalty, the first case under the market access rule

SEC press release 2013-222: "the SEC's first enforcement action under the market access rule". Knight settled "without admitting or denying the findings".

CAUTION"It was just one technician who forgot to copy a file"

That was the trigger (paragraph 15). But most of the SEC order is about something else: no written deployment procedure, no second review, no output monitoring, no halt, no incident response (paragraphs 15, 21–27).

CAUTION"They lost $440 million in 45 minutes"

$440 million was Knight's first estimate. The booked trading loss was $457.6 million; the SEC says $460 million. Orders ran for 45 minutes, but the loss was locked in as positions were unwound afterwards.

CAUTION"There were 97 warning e-mails and nobody bothered to read them"

The 97 e-mails are real (paragraph 19). But the SEC says Knight "did not design these types of messages to be system alerts". The lesson is about alert design, not lazy staff.

CAUTION"The new code was buggy"

The new RLP code worked on 7 servers (paragraph 16). The danger was old code not deleted, a reused flag, and one missed server.

CAUTION"Knight had no kill switch at all"

SMARS had no procedure to halt on aberrant activity (paragraph 21) and there was no incident response procedure (paragraph 27). But Knight did have automatic shutdown for certain strategies of one trading group (footnote 7).

FALSE"A rogue AI algorithm"

There is no AI or machine learning anywhere in the record. It was old order-routing code doing exactly what it was written to do, woken by a reused flag on a server that missed an update (SEC, paragraphs 13–16).

FALSE"Knight went bankrupt that day"

On 6 Aug 2012 Knight received $400 million from Jefferies, Blackstone, GETCO, Stephens, Stifel and TD Ameritrade, convertible into about 267 million shares at $1.50 a share. On 1 Jul 2013 Knight and GETCO combined to form KCG Holdings.

Source: SEC Order 34-70694 (paragraphs as noted in each row); SEC press release 2013-222; Knight releases of 2 Aug, 6 Aug and 17 Oct 2012.

Part 5

Verdict

Scores are Mark's judgement based on the sources in Part 8, not data from a study.

Avoidable
5/5
Warnings in advance
4/5
Transparent inquiry
4/5
Lesson retained
3/5
Done right afterwards
  • Removed the faulty software, disclosed the loss the next day
  • Raised rescue capital within 5 days
  • The SEC published a detailed record that set the standard for a new rule
Not done right
  • The 2011 incident got only a narrow fix
  • Compliance paperwork said "compliant" when it was not
  • A no-admit settlement makes responsibility hard to pin down

No villain, no AI. Just a deployment process missing every safety net, meeting an order machine faster than any human.

More: why this is a lesson for every software team

Take away the word "stocks" and the rest is everyday life on any software team: an old feature nobody deleted, a config flag reused to save effort, an unchecked rollout to several machines, an error-mail channel everyone filters into a folder.

What made Knight different was speed. An automated order machine can do in minutes what people would need a week to do. That is why the SEC stresses that controls must be as automatic as the system itself, and that stopping the system must be prepared in advance, not invented mid-incident.

Source for "More": SEC Order 34-70694, paragraphs 3–4 (the role of controls in automated trading); the rest is Mark's commentary.

Part 6

Work lesson: deploy with a checklist and a stop button

For every "push to production": software updates, config changes, mass e-mails, ad campaigns.

Four steps for every deployment: automate and verify every target, delete dead code and never reuse a flag, design real alerts and compare output with input, prepare a stop button and a written incident procedure. 1 · verify every target"8/8 same version"automated, second reviewer 2 · delete dead codenew flag, new namenever reuse an old flag 3 · alerts are alertscompare out with inlimits that block 4 · stop button, writtenwho presses, whenstop first, fix later "I'm sure it got copied everywhere" is not evidence that it did.
Each step patches one failure of 1 August. A missed server, a reused flag, unread e-mails, and a rollback invented mid-incident.Source: failures from SEC Order 34-70694, paragraphs 13–15, 19, 21–27; the steps are Mark's suggestions.
A checklist with a name on itList every target. After the deploy, someone else confirms each one runs the right version.
Delete what you don't useCode, flags, accounts, access left idle for nine years are nine years of risk waiting.
Fewer alerts, real alertsA channel that gets 97 e-mails nobody reads is no channel. Cut the noise; keep only what should wake someone up.
Practise stopping before you need toWrite down who may stop it, how, and when. Don't "roll back" until you know the old version is safe.
Without appropriate controls, automation can turn an otherwise manageable error into an extreme event.SEC, Order 34-70694, paragraph 3 (2013), paraphrased.
Part 7

Read next

SEC, Order 34-70694 (2013)Short, and reads like an incident analysis. Paragraphs 12–27 are the core.
Knight's releases of 2 and 6 August 2012Watch a firm describe its own incident within 24 hours, then raise rescue money within 5 days.
Google, Site Reliability Engineering (2016), chapters on releases and incident managementHow large teams do staged rollouts, safe rollbacks and incident procedures.
Anatomy of a Disaster #06: Titanic (1912)When regulation lags technology, and why to plan for the worst case.
Part 8

Sources

Primary documents first: the regulator's order and the company's own SEC filings.

  1. SEC, In the Matter of Knight Capital Americas LLC, Exchange Act Release No. 70694, Admin. Proc. File No. 3-15570, 16 Oct 2013, Section III, paragraphs 1–51. sec.gov
  2. SEC, press release 2013-222, "SEC Charges Knight Capital With Violations of Market Access Rule", 16 Oct 2013. sec.gov
  3. Knight Capital Group, release of 2 Aug 2012, Form 8-K, Exhibit 99.1. sec.gov/Archives
  4. Knight Capital Group, "Knight Capital Group Completes $400 Million Equity Financing Agreement", 6 Aug 2012, Form 8-K, Exhibit 99.1. sec.gov/Archives
  5. Knight Capital Group, Q3 2012 results release, 17 Oct 2012, Form 8-K, Exhibit 99.1. sec.gov/Archives