Knight Capital left retired code on the server that reached the market

A retired function is still a function if something can call it. The dangerous day is not the day a firm stops using it. It is the later day, when a new switch wakes the old function on a machine the new code never reached, and the market is already open.

The Securities and Exchange Commission's order of 16 October 2013, Release No. 70694, institutes proceedings against Knight Capital Americas LLC. Knight consented for those proceedings without admitting or denying the findings, except the Commission's jurisdiction, which it admitted. The order concerns a rule requiring a broker or dealer to control the risks of market access, including an automated system that does what the firm did not mean. The morning is 1 August 2012.

Knight was a US broker-dealer and a FINRA member, based in Jersey City. Until 1 July 2013 it belonged to Knight Capital Group. That group then combined with GETCO Holding Company to form KCG Holdings, and the order describes Knight as KCG's wholly owned subsidiary. Through 2011 and 2012 its trading generally represented approximately ten percent of listed US equity volume. SMARS, the router in these findings, generally represented approximately one percent or more.

SMARS receives parent orders and sends child orders out for execution. Knight was changing it so customers could use the New York Stock Exchange Retail Liquidity Program, due to begin on 1 August 2012. The new code was meant to replace unused code called Power Peg, which Knight had stopped using years before. Power Peg was still present and could still be called. The new code reused the flag that used to turn Power Peg on. Knight intended to delete Power Peg, so a flag set to "yes" would run the new code rather than the old function.

In the old function, a cumulative count stopped child orders once the parent was filled. Knight ceased using Power Peg in 2003. In 2005 it moved that count earlier in the SMARS sequence and did not retest Power Peg to see if the old function would still behave if called.

Deployment began on 27 July 2012, a few servers at a time. One technician did not copy the new code to one of eight SMARS servers. No second technician reviewed it. Nobody realised the eighth server still had Power Peg and did not have the new code. Knight had no written procedure requiring that review.

On 1 August the seven updated servers handled Retail Liquidity Program orders correctly. Orders with the reused flag, on the eighth server, triggered defective Power Peg. Because the cumulative count had been moved, that server sent child orders in a rapid sequence, ignoring shares already filled. Another part of the platform knew the parents were complete. It did not tell SMARS.

For those 212 parent orders, SMARS sent millions of child orders: 4 million executions in 154 stocks, more than 397 million shares, in approximately 45 minutes. Knight took an approximately $3.5 billion net long position in 80 stocks and an approximately $3.15 billion net short position in 74 stocks. It lost more than $460 million on positions it had not wanted, and it hit significant net capital problems. Many of the orders were short sales. Knight did not mark them short, as Rule 200(g) of Regulation SHO requires, and it did not meet the locate duty in Rule 203(b).

The executions moved prices. In 75 stocks, Knight was more than 20 percent of volume and contributed to moves of more than five percent. In 37 of those, the move exceeded ten percent and Knight was more than 50 percent of volume. Some other participants got worse prices than they otherwise would have, and some got better prices.

A warning came before the open and was not used. Separate pre-market orders caused an internal system, from approximately 8

a.m. Eastern Time, to email a group of staff. Ninety-seven messages, called BNET rejects, named SMARS and an error described as "Power Peg disabled". All of them arrived before the 9 a.m. open. Knight had not designed them as alerts, and people generally did not read them. The order says they were a chance to find the fault before the open. They were not acted on, and they were not used to diagnose it afterwards.

The live response made it worse. Knight had no incident procedure telling people what to do. The technology team hunted for the fault while SMARS kept sending millions of child orders. In one attempt they removed the new code from the seven servers where it had been deployed correctly. The order says that worsened the failure: further parent orders then activated Power Peg on those servers too.

Controls existed, but not at the last step. Interfaces upstream of SMARS could block a bad order. SMARS itself lacked a sufficient comparison of orders going out against orders coming in, and Knight had no procedure to halt SMARS when SMARS was the thing misbehaving. A price cap, 9.5 percent from the national best bid or offer at the moment the parent arrived, did not cover a smaller move. It also did not apply to the 212 orders, which were received before the open for the opening auction.

Position limits on some desks ignored orders still open. There was no firm-wide capital threshold linked to an automated stop. The 33 Account, which held executions the systems could not match to a parent, had a $2 million gross limit and was not tied to any automated firm-wide control. It filled with an unusual position, and because positions from several sources sat together, staff could not quickly see what they were. PMON, the main risk screen, watched positions only after execution. It did not stop orders, did not alert and did not display the limits. In this volume it lagged, and its figures were inaccurate. One trading group's strategies did shut off when P&L limits broke. That switch was not in front of SMARS.

An earlier loss had invited a wider question. In October 2011 a weekend disaster-recovery test left the lead market-making desk quoting from test data on Monday morning. The loss was nearly $7.5 million. Knight limited that system to market hours, stopped it quoting after an execution and added a checklist line for the test data. It did not ask whether any system could emit a bad order, for any reason, and it had no test for stale data. The order says the firm kept answering the exact miss in front of it.

The paper trailed the routers. Before the 14 July 2011 compliance date, with some duties extended on 30 June 2011 to 30 November 2011, staff concluded existing controls were enough. They inventoried what they had. They did not examine a SMARS malfunction, or PMON's inability to block an order. The written description stayed incomplete for months. The September 2011 narrative omitted the Electronic Trading Group, a large share of the firm's orders. The lead market-making desk was added only in July 2012, nearly a year after the description was due.

In March 2012 the CEO certified that Knight had "processes" to comply. He did not certify that the controls complied. The order calls that a drafting error, not an intentional one. He did not notice it, and he believed he was certifying compliance. Eight senior staff had signed a similar error and believed the firm complied. Certifying that processes exist, the order says, is not certifying that controls are reasonably designed and comply with the rule. Review procedures were also unclear, and they never asked about deployment or about unused code left on servers.

The order is plain about the release. Other groups had written deployment procedures. SMARS did not, and a second technician was not required. There was no rule to retest unused code after someone touched it. A written double-check of this deployment, the order says, could have shown the missed server and averted 1 August. Putting the BNET emails into monitoring could have prevented it as well. Retesting Power Peg in 2005 could have shown that the move had disabled the cumulative count inside that old code. Knight also needed clear guidance on when technology staff must disconnect a malfunctioning system. It had none. The router stayed on the market while people looked.

The Commission censured Knight and ordered it to cease and desist from violating Section 15(c)(3), Rule 15c3-5 and Rules 200(g) and 203(b) of Regulation SHO. The civil penalty was $12 million, payable to the Treasury within ten days. The Commission weighed remedial steps and cooperation after 1 August. An independent consultant, not unacceptable to the staff, must review the development lifecycle for business-critical systems and the controls on deployment, on routers, on firm-wide capital limits linked to orders before a trade and on incident response. Knight cannot dismiss that consultant without the staff's written approval.

The question I would want answered before the next release is the one the order keeps returning to. Who checked that the new code, and the removal of the old code, reached every server that can still send an order, and who may disconnect that server when the orders are no longer the ones anyone meant?

1. Require a second check, in writing, before retired code is allowed to stay callable. One person missed one server of eight. Nobody else looked. A note of the intention is not proof the eighth server matches the rest.

2. Remove retired code, or retest it if you touch it. Power Peg was dropped in 2003, altered in 2005 and left callable. Its old flag was handed to the new code. A function you no longer mean to run is still in production until nothing can call it.

3. Name someone who must read the early message before the open. Ninety-seven emails saying "Power Peg disabled" arrived from approximately 8

   a.m. They were not treated as alerts, and they were not used to find the fault. A message that names the retired function is already the incident.

4. Decide in advance who may disconnect the system. There was no such rule. During the search the router kept sending, and removing the new code from the healthy servers woke the old code there too. Stopping the system is an owned decision. It is not a side effect of debugging.

An agile session can be where the people who ship a change, and the person allowed to take the system off the market, write those four rules against one release that is actually waiting.