Cybersecurity

Supply Chain Attacks from SolarWinds to xz utils: Lessons Learned

Every organisation depends on software it did not write. Open-source libraries, commercial vendors, build systems, and update mechanisms form a vast, interconnected supply chain — and that supply chain is now a primary target for sophisticated attackers. Two incidents, four years apart, frame the problem with unusual clarity: the SolarWinds compromise, discovered in December 2020, and the xz Utils backdoor attempt, discovered in March 2024. Together they illustrate both the scale of the threat and the fragility of the ecosystem.

SolarWinds: The Stealthy Intrusion

SolarWinds Orion is a network-monitoring platform used by thousands of organisations, including large enterprises and government agencies. In the attack, suspected to be carried out by a Russian state-sponsored group, attackers compromised SolarWinds’ build environment and inserted a malicious backdoor — later named SUNBURST — into a legitimate signed software update, which was then distributed to roughly 18,000 customers.

The brilliance of the attack was its patience and restraint. The backdoor lay dormant for about two weeks after installation, then contacted command-and-control servers using benign-looking domain names. The attackers were highly selective, focusing on a small number of high-value targets rather than mass exploitation. This made detection extraordinarily difficult: the compromised update was signed, legitimate SolarWinds code, and the malicious activity was low and slow.

The breach affected multiple U.S. federal agencies, including departments of Treasury, State, Commerce, and Homeland Security, as well as private firms. The full scope may never be known. The U.S. government publicly attributed the attack, and the incident triggered a sweeping review of software supply chain security.

The Response to SolarWinds

The SolarWinds attack catalysed change. In May 2021, the U.S. issued Executive Order 14028 on Improving the Nation’s Cybersecurity, which included supply chain requirements. Among the most significant was a requirement that software sold to the federal government provide a Software Bill of Materials (SBOM) — a machine-readable inventory of components, mirroring an ingredients list on food packaging. The National Institute of Standards and Technology (NIST) was directed to develop secure software development standards (the SSDF).

The SBOM concept spread beyond government procurement. Increasingly, enterprises demand SBOMs from vendors, and standards such as SPDX and CycloneDX have matured. The idea is simple but powerful: you cannot secure what you cannot see, and you cannot respond quickly to a vulnerability in a component if you do not know it is there.

xz Utils: The Near-Miss

Four years later came a different kind of attack, and in some ways a more alarming one. In March 2024, a Microsoft engineer investigating slow login times on Debian systems discovered a subtle backdoor in xz Utils, a widely used compression library. The backdoor, inserted over a period of years by a maintainer going by the name “Jia Tan,” was engineered into the build process so that it would only activate under specific conditions — notably when linked into an OpenSSH server patched by a specific distribution. Had it been widely deployed, it could have enabled remote compromise of systems across the internet with a key only the attacker held.

The attack was caught largely by luck — by the meticulousness of one engineer and a timing shortfall. But the details were chilling. “Jia Tan” had spent years building trust in the xz project, submitting legitimate patches, becoming a co-maintainer, and being supported by a cast of sock-puppet accounts that pressured the original maintainer to hand over more control. The objective was not a single vulnerability but the compromise of a project deeply embedded in the Linux ecosystem.

What xz Revealed

The xz attempt exposed the stark imbalance at the heart of the open-source ecosystem. Critical infrastructure depends on software maintained, often, by a small number of unpaid, overworked volunteers. The maintainer of xz Utils was essentially a one-person operation carrying a burden the entire internet relied upon. This is not unique: numerous critical projects are maintained by tiny teams or single individuals.

The incident also showed the power of “social engineering the supply chain” — not just attacking code, but attacking the trust relationships and social processes by which code is maintained. The attacker’s patience was extraordinary, and the strategy of legitimate, long-term contribution followed by a malicious insertion is difficult to detect because it exploits the very norms of open-source collaboration.

Broader Context

SolarWinds and xz are the marquee examples, but the pattern is broad. The 2017 NotPetya attack spread through compromised Ukrainian accounting software and caused an estimated $10 billion in damages globally — one of the most destructive cyber events ever. The 2021 Log4Shell vulnerability, in the ubiquitous Log4j logging library, was exploited within hours of disclosure and remains exploitable in unpatched systems years later. The 2024 MOVEit and 2023 3CX incidents targeted widely used third-party tools.

The common thread is that attackers increasingly find it more efficient to compromise a trusted supplier or library once and thereby reach many victims, than to attack each target individually.

Lessons and Countermeasures

The response has several components:

  • SBOMs and dependency visibility: Knowing what software contains what components, and being able to respond when a component is vulnerable.
  • Build and signing integrity: Reproducible builds, signed commits, and secure build environments reduce the risk of tampering in the development pipeline.
  • Dependency and update hygiene: Pinning dependencies, vetting updates, and controlling the automatic distribution of third-party code.
  • Investing in upstream: Funding and supporting the maintainers of critical open-source projects, so that single points of failure are reduced.
  • Zero-trust segmentation: Limiting what a compromised component can reach.

The Broader Pattern of Infrastructure Attacks

SolarWinds and xz are the most prominent examples, but they sit within a larger pattern of attacks on the shared infrastructure of the internet. The 2021 attack on Codecov’s Bash Uploader compromised a tool used by thousands of development teams and exposed credentials. The same year, attackers compromised a test environment at HPE’s Aruba Central, gaining access to customer data. In each case, a supplier trusted by many clients became the vector. The economics of attacking a supplier rather than individual targets are so compelling — spend effort once, gain access to many — that it has become a dominant strategy for well-resourced adversaries.

Reproducible Builds and the Verification Frontier

One of the most promising defences is the concept of reproducible builds: compiling software from source in a way that produces bit-for-bit identical outputs, so that anyone can verify that a distributed binary matches the published source. Several open-source projects — Debian, Tor, Bitcoin Core — have adopted reproducible builds, but adoption is far from universal. Cryptographic code signing helps, but it only verifies who built the binary, not whether the binary was tampered with at build time. The combination — signed, reproducible builds in an isolated, observed environment — is the ideal, but it is currently the exception rather than the norm.

Funding the Commons

The xz incident crystallised a problem that open-source developers have raised for years: the internet runs on infrastructure maintained by unpaid or underpaid volunteers. The Linux Foundation, Open Source Security Foundation (OpenSSF), and initiatives like the Sovereign Tech Agency in Germany have started to channel funding to critical projects and offer security audits and incident-response support. But the scale of dependence vastly exceeds the resources allocated. A widely cited survey from the Linux Foundation found that a meaningful fraction of widely used open-source projects had only a single maintainer. Closing this gap — through corporate funding, foundation support, or even direct public investment — is a structural necessity, not a charitable gesture.

The Detection Challenge

Supply chain attacks are intrinsically hard to detect because they exploit trust. A malicious commit in a popular library, or a backdoored dependency, may look like legitimate maintenance to any review. Automated tools — vulnerability scanners, dependency analysis, static analysis, and behavioural monitoring — help but are insufficient without human review. SBOMs help by making dependencies visible, but visibility alone does not catch a cleverly disguised attack. The xz attempt was discovered by a combination of a skilled engineer’s intuition, careful measurement of latency anomalies, and luck. That is not a scalable solution.

Regulation and Liability

Regulators are starting to move. The EU’s Cyber Resilience Act, agreed upon in 2024, mandates security requirements for products with digital elements, including vulnerability reporting and security support throughout the product’s lifecycle. In the U.S., the National Cybersecurity Strategy and subsequent implementation plans propose shifting liability to software vendors for insecure products — a fundamental change from the status quo. Whether regulation produces better security or merely compliance paperwork is debated, but the direction is toward greater accountability for the software supply chain.

Conclusion

Supply chain attacks are not new, but they have become the defining threat of the modern software era because they exploit the fundamental structure of modern development: dependence on trust, on third parties, and on shared code. SolarWinds showed that a patient attacker can weaponise a legitimate update. xz showed that even the open-source commons that underpins everything is vulnerable to a patient social engineering campaign. The lesson is blunt: trust in the supply chain must be verified, not assumed. That is easier said than done, but the alternative — discovering the next breach only after it has already spread — is worse.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button