What happened?
On October 1, 2020, a failure in the Tokyo Stock Exchange cash equity trading system, arrowhead, led TSE to suspend auction trading for all issues for the entire day.
The incident began with a memory-card malfunction in the shared storage system, or NAS. The NAS used two units so that processing could switch automatically from No.1 to No.2 if one unit failed. That automatic switchover did not occur.
The primary source for this article is the Investigation Report submitted on November 30, 2020 by the Investigation Committee of Independent Outside Directors in Relation to the System Failure. This English article cites JPX’s official English translation, published later by JPX. The report’s account of events relies on materials and information provided by JPX and TSE; the Committee’s main purpose was to evaluate their views and findings and make recommendations.
Nine passages that stood out
Page references below are to the page numbers printed in the official English translation.
Automatic and manual switchover attempts failed. Staff then prepared to physically detach cables
According to the account provided by JPX and TSE, after the automatic switchover from NAS No.1 to No.2 failed, TSE and Fujitsu tried forced power-off, transfer of control, port blocking, and another command with changed conditions. None produced the desired switchover.
The response then shifted to physically detaching cables, with preparations made for equipment-maintenance staff to enter the server room.
While that was being arranged, Fujitsu continued trying commands that could force NAS No.2 to seize control. At 09:26, adding options to a command attempted three minutes earlier finally produced the switchover.
The requirement was a switchover within 30 seconds. The actual setting prevented automatic switchover
TSE’s requirements for the third-generation arrowhead called for NAS-related processing to continue after an equipment malfunction with a switchover completed within 30 seconds.
The “On Panic” parameter, however, was set to False. TSE understood that setting to disable immediate takeover while leaving standard takeover available.
Under the actual specification, receiving a panic notification first while the parameter was False prevented both immediate takeover and standard takeover.
As a result, processing did not switch automatically from the failed NAS No.1 to NAS No.2.
The default changed from False to True, but the description of what False did stayed old
For the generation used with the first arrowhead, the manual and the actual NAS behavior matched: with On Panic set to False, standard takeover still worked.
The product specification later changed so that standard takeover would not start under a particular failure sequence when False was selected. The default value was changed at the same time, from False to True.
The supplier’s specification document recorded the change in the default value, but did not revise the description to reflect the changed behavior under False.
According to Fujitsu’s report, it did not recheck the False behavior on actual equipment and treated the documentation review as sufficient.
Fujitsu had received a revised manual about 11 months before the failure
The Committee identified the mismatch between the Fujitsu NAS manual and the actual specification as central to the failed automatic switchover.
The manual was not still uncorrected when the October 2020 failure occurred.
The Committee states that Fujitsu had received a revised version of the NAS manual in November 2019.
The Committee said that if the revised manual had been examined in detail and the erroneous setting recognized at that point, the setting could have been changed.
Switchover testing had been done before launch, but not under the failure condition that actually occurred
TSE and Fujitsu had tested NAS switchover behavior before full-scale deployment. A heartbeat-communication cut-off between the two NAS units had been used to confirm that operations could continue.
The October failure was different: loss of functionality in the NAS unit produced a panic notification before the heartbeat messages stopped.
The pre-launch tests did not reproduce a loss of functionality in the NAS unit under the conditions that occurred in this failure.
The Committee said such a test could have revealed the defective setting. It separately said that testing the arrowhead setting values before shipment could have exposed the defect, and that TSE should arguably have required Fujitsu to perform such tests given the NAS’s importance.
Even the emergency trading-suspension function depended on the same failed NAS
arrowhead had an emergency trading-suspension function for cases in which normal trading suspension and market suspension could not be used.
But multiple trading-suspension functions relied on files stored on the NAS.
The emergency function used a file on the failed NAS to designate which issues should be suspended, so it could not be used either.
The Committee said an emergency function intended for failures elsewhere should have been configured so it could be executed with complete reliability in any situation.
Trading was 'suspended' by cutting the network. Execution processing continued inside arrowhead
Because the normal and emergency suspension routes were unavailable, TSE cut off the load balancer connecting arrowhead with trading participants.
That stopped network communication, but it was not an instruction to stop execution processing.
Orders accepted before the load-balancer cut-off continued to be processed for execution inside arrowhead after the network was cut.
The resulting execution notices were retained in the system instead of being sent to trading participants. For about one minute after 09:00, execution data was also distributed through some market-information services before TSE notified users that the executions were invalid.
Only about 38% by trading-value share said they could resend orders — all were overseas securities firms
After the NAS switchover succeeded, TSE considered rebooting arrowhead and resuming trading that same day. Trading participants, however, would have needed to handle orders in an unusual way, including resending orders that still appeared as already submitted in their own systems.
According to JPX and TSE’s account, TSE gathered opinions from participants and vendors. The share cited here is based on participants’ past trading value, rather than their share of the number of participants.
Participants that said they could resend orders represented only about 38% of trading value, and they were limited to overseas securities companies.
The report also says resuming trading after rebooting arrowhead had not been tested even in the test environment, making the outcome impossible to predict.
The Committee called overemphasis on 'Never Stop' putting the cart before the horse
arrowhead had operated under the slogan “Never Stop,” with reliability treated as a central goal. The Committee did not reject that goal.
It did, however, say that treating order acceptance from 08:00 and trading from 09:00 as absolutes even after the failure made smooth resumption harder after recovery, describing that response as “putting the cart before the horse.”
The report says planning should start from the realistic assumption that “unacceptable events will always occur.”
It argued that response planning must therefore consider how to limit the impact and resume quickly and safely, rather than focusing only on preventing any interruption.
What the report says about the background
The Committee did not identify the memory-card malfunction itself as a failure of TSE’s response. The card was still within its service life, and the event was treated as an accidental hardware failure that could not be eliminated entirely.
The larger problem was that the redundant NAS did not perform the automatic switchover intended to contain that failure. The Committee placed heavy responsibility on Fujitsu for the incorrect manual and resulting setting, while also saying TSE bore responsibility to a certain extent because testing before launch had not reproduced the relevant failure condition.
The same report also found insufficient preparation for what would happen after that first safeguard failed: forced manual switchover, emergency trading suspension without the NAS, and resuming trading after an abnormal suspension. Its final emphasis on resilience alongside “Never Stop” follows from those layers of fallback that did not work as intended.
Corrections to this article
We distinguish the companies’ accounts, supplied to the Committee, from the Committee’s assessments (pp.4–5). Note 04 attributes receipt of the revised manual to the report. Note 05 separates pre-launch failure simulation from the assessment of testing arrowhead settings before shipment and cites the continuation on p.24. Notes 07–08 expand the source references; note 08 clarifies that roughly 38% refers to past trading-value share, rather than the number of participants.