What happened?
On February 28, 2021, part of Mizuho Bank’s ATM network became unavailable, and large numbers of customers had passbooks or cash cards retained inside ATMs instead of being returned. Separate system failures followed on March 3, March 7 and March 12.
Mizuho Financial Group established a System Failure Special Investigative Committee made up of outside experts. The committee examined not only the technical causes of the four failures but also how warning signs were handled, how customer impact was understood, weekend communication arrangements, business continuity planning, lessons from earlier failures and aspects of corporate culture. Its public report, released June 15, 2021, is a 167-page Japanese PDF.
The source report used for this article is in Japanese. English renderings of Japanese wording below are translations or summaries by From Reports, not an official English translation of the investigation report.
Background overview: Mizuho Bank (Wikipedia)
Eleven passages that stood out
The report is remarkably specific about what happened after the ATMs failed: which warnings were missed, how errors were passed along on paper and by phone, and how slowly important information reached customers waiting beside the machines. These eleven notes make the operating practices visible even without specialist IT knowledge.
The warning threshold was 80%. Usage reached 87% the day before the failure, but no action was taken
In the February 28 incident, utilization of the relevant index file (INDEX FILE) had been rising.
On February 27, the day before the failure, it had reached 87%.
The warning threshold was 80%, and a warning was being written to the system log.The responsible section either missed the warning or did not recognize it as requiring action. A separate operations section also received the warning, but special instructions said it did not need to respond during this work. Its manual also put the telephone notification on the next business day, so the warning was not relayed to the responsible staff member.
The committee treats the handling of these warnings as one of the missed opportunities to prevent the incident.
Errors were watched on screen, printed on paper, handed over, then read out by phone
A large volume of error messages appeared during the morning of the incident.
Monitoring staff scrolled through the display and visually checked the messages, then printed errors they judged important and handed the paper to an operator.
The operator then called the responsible staff member and verbally relayed the information from the printout. The recipient was not required to record the error ID, and there was no call memo or recording that would allow the exchange to be reconstructed later.
The committee notes that this made it impossible to verify afterward whether key error information had been communicated accurately.
The 'emergency first report' also said the monitoring system showed ATMs operating normally
At 10:15 a.m., an “emergency first report” email was sent to related departments. It said ATM transaction failures were occurring frequently and that the cause was unknown.
At the same time, the message included standardized wording to the effect that the ATM monitoring system showed normal operation.
Staff in the headquarters departments that received the email did not move from that message into concrete customer-response actions.The committee says the wording may have reinforced a normalization bias among recipients.
Even after staffing rose to 14 people, roughly nine out of ten calls still went unanswered
On holidays, the ATM centers in Tokyo and Osaka together were staffed by eight people. As reports of retained passbooks and cards surged, managers joined the response and the number handling calls rose to 14.
Even so, the call-loss rate remained in the 90% range after 10 a.m.
A separate call center had 32 people working, but during some periods roughly two-thirds to seven-tenths of calls there also went unanswered.Public relations asked for confirmation. With no reply, it drafted the customer notice itself
The public-relations function wanted to post a notice about the failure on the bank’s website and emailed related departments for a status update.
When no response came, public relations drafted the external notice itself.
After the relevant departments confirmed they had no objections, a formatting error forced the upload process to restart. The first notice about unavailable services appeared at 1:15 p.m.Later, as the scale of retained passbooks and cards became clear, the question arose whether the bank could tell customers that their items would be returned later and that they could leave the ATM. Public relations again asked the related sections for confirmation, received no explicit response, and again drafted wording on its own.
The notice saying the bank would contact customers and return their passbooks or cards later was confirmed at a department-head meeting around 3:40 p.m. and posted at 3:58 p.m.
ATMs retained 5,244 passbooks and cards. Only 1,244 were returned that day
During the February 28 failure, ATMs retained a total of 5,244 passbooks and cash cards belonging to 5,170 customers.
Mizuho used remote operations, security companies and branch staff to retrieve them, but
only 1,244 items could be returned on the day of the incident.
The remaining 4,000 were returned later by mail, at branches, by visits to customers and other means.Most had been returned by March 7, but the final return was not completed until April 22.
Customer reports included seven-hour waits, 196 phone attempts and a missed examination
The report records concrete complaints that customers submitted to the bank.
They included a customer who said they waited more than seven hours after a passbook or card was retained, another who incurred parking fees while waiting, and
a customer who said they called another bank contact point 196 times before finally getting through.
Other reports included missing an important examination, taking time off work, missing an event, spending hours in the cold without food or drink, and being unable to use a restroom for a long period. These are reports made by customers and recorded in the committee report; they should not be read as independent factual findings by the committee in every individual case.
The committee estimated that a later ATM specification change could have prevented 4,915 of the 5,244 retained items
Mizuho Bank’s ATMs were designed to retain a passbook or card inside the machine under a relatively broad set of error conditions. After the MINORI core banking system went live, substantial retention incidents had already occurred more than once, including in 2018 and 2019.
After the February 28 incident, Mizuho changed the specifications to return passbooks and cards for account transactions in cases of communication failures and ATM processing-section errors. The report describes the changes in place as of May 9, 2021.
The committee estimated that, had this specification already been in place, 4,915 of the 5,244 retention cases would not have occurred.
The committee says earlier incidents had provided opportunities to reconsider the specification itself, but that review had not been carried through sufficiently.
Investigation committee finding
Investigation committee's account of ATM service restoration (site translation from the Japanese original)
Meanwhile, the ATM center continued its usual procedure: it remotely returned retained passbooks and cards to customers who managed to reach the center by phone and restored the machines. But when the next customer used one of those ATMs, the machine sometimes retained their passbook or card again.
By 12:30 p.m., staff had decided customers could leave. The message did not reach the people waiting at ATMs
At around 12:30 p.m., the call center adopted a policy of telling customers whose passbooks or cards had been retained that the items would be returned later and that they did not have to remain at the ATM.
That information, however, did not spread at the same time to the ATM center, branches and the website. Customers continued waiting at ATMs without knowing when their items would be returned.
The committee says timely communication could have substantially reduced customer waiting time. It also notes that system restoration was prioritized in the morning, while information such as the number of stopped ATMs and retained cards or passbooks was not being collected adequately as customer-impact data.
Drills for the holiday system-failure plan, BCP 736, had been omitted for five years
There was no business-continuity plan specifically built around a mass retention of passbooks and cards. There were, however, existing BCP arrangements for situations such as ATMs becoming unavailable because of a system failure on a holiday.
Drills for the holiday system-failure plan, BCP 736, had been omitted for the previous five years.
The bank had instead conducted drills for BCP 46, applying a rule allowing a drill to be omitted when its content was the same as another drill. Training on the ATM-center side was also largely formal, such as checking that an emergency email could be sent and received.The committee also found departments where only one emergency contact was registered, and others where multiple contacts were listed but some information had not been kept current.
In the committee's analysis, saying 'I did what I could in my own area' could become the rational organizational choice
Near the end of the report, the committee moves beyond technical causes and addresses corporate culture.
It says the organization showed weakness in acting proactively beyond narrowly assigned responsibilities during a crisis, especially when information was incomplete and taking a broader risk decision could later expose someone to responsibility.
The committee describes a structure in which avoiding that risk and being able to say “I did what I could within my own area” could become the rational choice inside the organization.
The committee adds an important qualification: the investigation alone could not logically prove a corporate culture in a strict sense. Even so, based on recurring similarities with earlier failures and the absence of decisive actions such as stopping ATMs, it says it was reasonable to infer an underlying culture that could encourage people to avoid proactive action for fear of making a mistake.
Investigation committee finding
Investigation committee's assessment of the head office's view of ATM users (site translation from the Japanese original)
Ordinary deposits, withdrawals and transfers had increasingly been directed away from branches and toward ATMs. Many retail customers therefore had contact with the bank only through an ATM and were, particularly from head office's perspective, 'customers without visible faces.' Could that have weakened head office's ability to respond as promptly and flexibly to customers as branches could?
What the report says about the background
The committee did not reduce the series of failures to isolated software defects. It grouped the broader issues around organizational capability in a crisis, IT-system governance, the customer perspective and corporate culture.
In the February 28 incident, a warning sign was missed, the mechanism for recognizing and escalating a flood of errors was weak, system restoration took priority over understanding customer impact, and weekend communication, BCP, public relations and branch operations did not connect effectively.
Mizuho had already experienced major system failures in 2002 and 2011 and had introduced prevention measures after each one. The committee nevertheless found that issues resembling earlier failures reappeared in 2021, including cross-department coordination, crisis decision-making, customer response and preparations for stable system operation.
Background source: report pp.122–130.
Corrections to this article
Note 05 now distinguishes the upload restart for the 1:15 p.m. first notice from the 3:58 p.m. later-return notice (pp.62–63). Note 07 labels customer complaints as testimony and removes an unsupported reference to compensation requests, and note 10 specifies that the holiday drills omitted for five years concerned BCP 736 (pp.83–84). Source pages were added to the background discussion, and the timing of branch actions was clarified. Because the publisher disallows embedding, source links now open the cited PDF page directly in a new tab.