What happened?
In September 2017, U.S. consumer reporting agency Equifax announced a major data breach. The final affected population grew to about 148 million people—nearly half the U.S. population and 56 percent of American adults, according to the report.
Attackers exploited an Apache Struts vulnerability in ACIS, an internet-facing system consumers used to dispute information in their credit files. They remained active in the Equifax environment for 76 days, from May 13 through July 30, 2017, and exfiltrated personal information.
The primary source for this article is the 96-page Majority Staff Report “The Equifax Data Breach”, released in December 2018 by the U.S. House Committee on Oversight and Government Reform. The report described the breach as “entirely preventable.”
Background: Equifax (Wikipedia)
Ten passages that stood out
A critical warning circulated to roughly 430 employees, yet scans failed to find the vulnerable software. Attackers used credentials stored in the system to access 48 unrelated databases. The report also records an expired monitoring certificate, confusing public-facing breach websites, and the questioning of individual employees afterward. These ten scenes follow the emails, testimony and practical failures behind the breach.
This article paraphrases and translates the source where necessary. It is not an official publication of the Committee or Equifax. Page numbers refer to the printed pages of the Majority Staff report, and evaluative conclusions are attributed to that report rather than presented as independent findings by From Reports.
About 430 people were told to patch within 48 hours, but the scan was not looking where the vulnerable Struts installation actually was
In March 2017, a critical Apache Struts vulnerability became public. Equifax circulated the alert to about 430 people and multiple distribution lists, telling those responsible for Struts installations that the issue was being actively exploited and required a patch within 48 hours under company security policy.
Security also ran an open-source component scan. It found no vulnerable Struts components.
Interim CSO Russ Ayres later explained that the scan had been run against the root directory rather than the subdirectory where Apache Struts was listed.
A separate scanner then checked 958 externally facing Equifax IP addresses twice and also failed to identify the vulnerable Struts installation.
ACIS needed only 3 databases for its job. Attackers were able to reach 48 unrelated ones
For its normal business function, ACIS needed access to only three databases in the Equifax environment.
But ACIS was not segmented from other, unrelated databases. After entering ACIS, the attackers found unencrypted application credentials—username and password—in a mounted file share.
They used those credentials to access 48 unrelated databases outside the ACIS environment.
According to Mandiant’s account cited in the report, the attackers ran about 9,000 queries across those databases, and 265 queries returned datasets containing personal information. That PII was not encrypted at rest.
A monitoring certificate was expired for 19 months. Once renewed, suspicious traffic appeared almost immediately
An SSL Visibility appliance sat in the path between ACIS and the internet so encrypted traffic could be decrypted for inspection by intrusion-detection and prevention systems.
The certificate used by the appliance to monitor the ACIS domain had expired on January 31, 2016. Traffic was configured to continue flowing even with the expired certificate, but the security systems behind the appliance could no longer inspect that encrypted traffic. The report says Equifax lacked visibility into ACIS network traffic for 19 months.
On July 29, 2017, Equifax loaded new certificates onto the appliance.
Almost immediately, the Countermeasures team detected a suspicious request from an IP address originating in China.
To identify victims, investigators first had to work out who owned the databases because no owner list existed
After the breach was discovered, Mandiant had to determine what data the attackers accessed and which consumers were affected.
That required working with the internal owners of the databases so investigators could understand data that was not clearly identifiable.
But, according to Mandiant’s account cited in the report, there was no list of Equifax database owners.
Before Mandiant could analyze some of the data, it had to identify and verify who owned the relevant databases.
The breach-response site looked suspicious, and Equifax's official Twitter account directed customers to another site for nearly two weeks
Equifax built equifaxsecurity2017.com in roughly three weeks to tell consumers whether they were affected and to offer protection services.
Because it was separate from the main equifax.com domain, security experts thought the link looked suspicious and confusing, according to the report.
The long name also confused Equifax employees.
The company’s official Twitter account reversed the word order and directed customers for nearly two weeks to a fake site, securityequifax2017.com, created by a security researcher.
The report calls it a phishing website. However, this researcher-created site showed visitors a warning when they tried to submit personal information.
Original figure reproduced from the report
Original figure reproduced from the report
The same consumer could get conflicting answers on a computer and a phone about whether the breach affected them
The real breach-response site had problems too. Some people could not enroll in credit monitoring or received error messages.
In other cases, consumers received conflicting answers about whether they were affected depending on whether they visited from a computer or a mobile phone.
Graeme Payne described a bottleneck between the large cloud-hosted front end and Equifax’s internal systems, which could process only a limited number of registrations at a time.
He compared it to filling a bathtub quickly while the water could leave only in drips: the intake capacity existed, but the internal processing path could not keep up.
A senior executive was fired for not forwarding the alert. He testified that nobody had told him to forward it
On October 2, 2017, Equifax terminated Graeme Payne, the Senior Vice President and CIO for Global Corporate Platforms who managed the ACIS environment. Human resources told him the reason was his failure to forward the Apache Struts alert email.
The next day, former CEO Richard Smith repeatedly referred in congressional testimony to an individual whose failure to communicate the need to patch was a human error behind the breach.
Payne told the Committee that he was one of about 430 recipients, that he was copied for information, and that he was neither the system owner nor the application owner under the patch policy.
He testified that nobody had directed him to forward such emails.
The Majority Staff report went further and characterized the termination as appearing public-relations motivated. That is the report’s assessment, not an independent finding by From Reports.
Struts was found on ACIS in January, but not tracked; its presence surprised staff in July
A 2015 internal audit had already identified weaknesses in asset and patch management. Yet as of July 2017, Equifax did not have a comprehensive, up-to-date inventory of its IT assets or the software running on them.
The report says Equifax employees had actually identified Apache Struts on ACIS in January 2017 while remediating a different Struts vulnerability.
The company did not document and track that information. When Apache Struts was found in the ACIS environment again in July, the report says its presence came as a surprise.
A system built in the late 1970s was still running in 2017. Equifax was 'lucky' the original developers were still there
ACIS had been built in the late 1970s to support consumer disclosures and disputes. After many modifications over the years, it was still running when the 2017 breach occurred.
Payne testified that one risk of continuing to operate the aging system was the dwindling number of employees who understood it. Equifax was, he said, “lucky” that the original ACIS developers were still on staff.
When Committee staff asked how many original developers that meant, the answer was “A couple of people.”
An email warned attack logs could be lost; the security chief said retention might suffice
In an August 17, 2017 email to Susan Mauldin and others, Francis Finley listed ACIS web-server logging as “Security Concern 4.” The retention figures given were 14 days and 30 days online; the report does not explain how those two periods relate.
August 17, 2017 — Francis Finley’s security warning email (excerpt)
making it difficult, to impossible, to reconstruct any malicious activityAsked whether 14 days / 30 days could be sufficient for the internet-facing ACIS environment, then-CSO Susan Mauldin said retention depends on the use and environment, and answered that it “certainly could be sufficient.”
What the report says about the background
The Majority Staff report did not reduce the breach to a single employee failing to forward an email. It described failures spanning patch execution, inventories of IT assets and software, expired certificates, network segmentation, the division of responsibility between IT and Security, and continued dependence on legacy systems.
At the time of the breach, the report says Equifax had allowed at least 324 SSL certificates to expire, including 79 associated with devices monitoring highly business-critical domains. A 2015 internal audit had also identified problems in asset and patch management before the 2017 incident.
Corrections to this article
Added the report’s context that the researcher-created fake site displayed a warning when visitors tried to submit personal information. Clarified that “14 days, and 30 days online” was a concern recorded in an internal email after the breach, and that the report does not explain how the two retention periods relate. Also made the attribution of the forensic firm’s account and the Majority Staff’s assessment clearer.