As the maintainer of Rust in openSUSE I am often asked to support build reproducibility in our supply chain. I've spent countless hours researching the problem, and discussing it with security experts to understand details.
Thanks to the XZ incident this topic has once again come up, since after any security incident people always use the attention to further their own agendas. As a result, I'd like to write my very not scientific thoughts about reproducible builds.
Before We Begin
Let's perform a small exercise. The next line describes my summary of reproducible builds.
Reproducible builds have no impact on the security of a supply chain, and come at a significant cost in time, energy and resources.
Now stop. What's that feeling in your body? What emotions are you experiencing right now?
If you agree with the above you might be feeling vindicated or positive toward me, the author.
If this is your first time hearing about reproducible builds maybe you're feeling curious to read more and see what I write next.
However if you're highly invested in the topic, you likely feel anger or frustration. "How could this person be so stupid! Why can't they see how good and correct this is!".
The strength of that reaction is proportional to your own emotional investment in the topic. I know this, because it happens to me just as much as anyone. Being a white man myself, something that we often don't talk about in our society is that many of us have been conditioned to believe our emotional decisions and thoughts are logical ones. This is probably tied with those feelings you're experiencing right now.
It's likely that some of our dear readers will have strong emotional connections to the topic we are about to discuss. This emotional investment clouds and blocks scientific and logical discussion of the situation. We need to put that to the side and focus on evidence, not our feelings.
Scientific Method
"Science is a rigorous, systematic endeavor that builds and organizes knowledge in the form of testable explanations and predictions about the world."
If we are going to have a scientific discussion about reproducible builds, we need to define our predictions (hypothesis) but importantly we must also define the null hypothesis.
Which would mean that our hypothesis is:
- Reproducible builds would have impacted or prevented historical and current threats against supply chains.
For the remainder of this blog we will consider that the null hypothesis is:
- Reproducible builds would not prevent historical or current threats against supply chains.
What is a Software Supply Chain?
A software supply chain is everything in between a developer typing out code on their machine to that application running on a consumer machine. There are a huge number of steps, and they vary from project to project. For the sake of this discussion, I'll describe the supply chain of a hypothetical application that is being submitted to openSUSE.
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
┌───────────────────┐
│ │ Developer │ Developer Workstation │
└───────────────────┘
│ │ │
Writes Code
│ │ │
▼
│ ┌───────────────────┐ │
│ Workstation │
│ └───────────────────┘ │
│
│ Submits │
┌ ─ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
▼ │
│ ┌───────────────────┐
│ Pull Request │ Code Repository │
│ └───────────────────┘
│ │
│ Merge
│ │
│ ▼
┌───────────────────┐ │
│ │ Git Main │───Build and Test────┐
└───────────────────┘ ▼ │
│ │ ┌───────────────────┐
Publish │ CI/CD │ │
│ │ └───────────────────┘
▼ │
│ ┌───────────────────┐
│ Release Artefacts │ │
│ └───────────────────┘
│ │
├ ─ ─ ─ ─Submits─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
│ │
│ ▼
┌───────────────────┐ Build Service │
│ │Open Build Service │
└───────────────────┘ │
│ │
┌ ─ ─Begins Building─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ │ │
│ ▼ Offline │
│ ┌───────────────────┐ Build Host │
│ │ Signature Checked │ │
│ └───────────────────┘ │
│ │ │
│ Unpacks │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ Build Application │ │
│ └───────────────────┘ │
│ │ │
│ Packages │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ RPM │ │
│ └───────────────────┘ │
│ │ │
│ Signs │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ Signed Package │ │
│ └───────────────────┘ │
│ │ │
┼ ─ ─ ─Distributes─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤
│
│ ▼ Package Mirrors │
┌───────────────────┐
│ │ Mirrors │ │
└───────────────────┘
│ │ │
┌ ─ ─ ─ ─Provides ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
│ │
│ ▼ Consumer Workstation
┌───────────────────┐ │
│ │Downloaded Package │
└───────────────────┘ │
│ Verifies
Signature │
│ │
▼ │
│ ┌───────────────────┐
│ Verified Package │ │
│ └───────────────────┘
│ │
│ Installs
│ │
│ ▼
┌───────────────────┐ │
│ │ Application Runs │
└───────────────────┘ │
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
Phew, that's a lot of steps to get from some code to a users machine! This is not even a complete or exhaustive diagram either, I'm sure there are more steps I've missed or could add.
The Attacker's View
Before we can discuss potential security improvements, we always need to discuss our attackers and their motivations.
When we are discussing something like supply chain security, the goal of an attacker is to have attacker controlled code running in the context of the consumers workstation. This could be for something as base as buttcoin mining, it could be to distribute malware, or it could be targetted to exfiltrate documents or access privileged government or corporate systems.
So what are some ways that an attacker could achieve this in our supply chain we demonstrated above? Each step on our diagram is a potential area to be attacked and compromised, and each of these has different methods and approaches.
I would hope that you can alredy see that there is a very large surface area here to defend, and that surface area has a lot of variation and complexity over it. Just from the top of my head, here are some potential attacks:
- Accidental or intentional addition of bugs into a code base causing code execution vulnerabilities.
- Malware on the developers workstation to allow injecting attacker code.
- Theft of the developers keys/tokens allowing them to submit code to the repository.
- Social engineering allowing an attacker direct access to merge code.
- Compromise of the code repository allowing injection of code to the repository.
- Theft of CI/CD tokens allowing code to be submitted to a repository.
- Alteration of release artefacts on the repository.
- Submitting altered release artefacts to a distro build service
- Addition of malicious patches or build scripts in the distro build service.
- Modification of sources/binaries during the package build process.
- Theft of the package signing keys allowing package forgery.
- Compromise of a mirror allowing malicious content to be served.
- Alteration of consumer workstation trust roots allowing attacker signed packages.
This is hardly exhaustive either. I'm sure that there are many more attacks that could be carried out.
Reproducible Builds
A reproducible build is where given the same source artefacts the resulting binary artefacts will always be byte for byte identical. It takes an astounding amount of work to achieve this because even variations in filesystem inode ordering, tarball generation, or even system clocks can alter a single bit of the build breaking the reproducibility.
What this allows is validation that if the source or binaries are altered before packaging and signing that these changes can be detected.
What Do These Protect?
So within our supply chain, what section is protected by these properties?
│ │
│ ▼
┌───────────────────┐ Build Service │
│ │Open Build Service │
└───────────────────┘ │
│ │
┌ ─ ─Begins Building─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ │ │
│ ▼ Offline │
│ ┌───────────────────┐ Build Host │
│ │ Signature Checked │ │
│ └───────────────────┘ │
│ │ ◀────────────────────────────────┼───┐
│ Unpacks │ │
│ │ │ │
│ ▼ │ │
│ ┌───────────────────┐ │ │
│ │ Build Application │ │ │
│ └───────────────────┘ │ │
│ │ │ │
│ Packages │ │
│ │ │ │
│ ▼ │ ├──── HERE
│ ┌───────────────────┐ │ │
│ │ RPM │ │ │
│ └───────────────────┘ │ │
│ │ │ │
│ Signs │ │
│ │ │ │
│ ▼ │ │
│ ┌───────────────────┐ │ │
│ │ Signed Package │ │ │
│ └───────────────────┘ │ │
│ │ ◀───────────────────┼────────────────┘
┼ ─ ─ ─Distributes─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤
│
│ ▼ Package Mirrors │
┌───────────────────┐
│ │ Mirrors │ │
└───────────────────┘
To add to this, to achieve this protection properly you actually need to build the package multiple times in parallel to compare the differences. This is discussed further in Tavis Ormandy's blog
This makes reproducible builds expensive from human time and labour to meticulously maintain the needed invariants of their software always being binary identical, it is expensive on resources requiring multiple parallel hardware/software stacks to isolate build processes, and it is expensive on energy as we require double to triple the electricity to compile things in duplicate/triplicate.
From our list above, what threats are mitigated by reproducible builds?
- Modification of sources/binaries during the package build process.
- Theft of the package signing keys allowing package forgery.
Historical Attacks
Lets now examine a number of recent attacks and how they were carried out, what could have mitigated these attacks, and if reproducible builds was one of those things.
XZ
The XZ incident was the discovery of a backdoor added to the XZ decompression library. This library was indirectly included in SSH processes on certain distros which would have allowed an attacker to remotely access a system.
This attack was carried out through an extended social engineering campaign. The backdoor was then inserted into autotools scripts and test artefacts.
Reproducible builds would not have detected this backdoor, since reproducing the build from the same compromised autotools outputs would have resulted in two identical compromised binaries.
Solar Winds
Solar Winds was subject to a supply chain attack in 2019-2020. The documented and suspected access vector was "the attackers might have compromised internal build or distribution systems of SolarWinds". A second attack was then discovered which relied on injected code into a http library.
In both cases it would appear that reproducible builds would not have detected this attack as the compromise of the build host and the injection of code would mean that hosts are reproducing the same compromised binaries in each case.
Other Attacks
The CNCF maintains a list of supply chain attacks. This is likely not comprehensive, but it gives us a good over view to analyse. There are 78 attacks here and they are helpfully categorised by attack type.
Within this set of attacks, only three may have been detected by reproducible builds.
Developer signing keys were stolen.
This application is not opensource, so users would be unable to rebuild and validate the results.
CCleaner would be able to prevent this by using HSM bound signing keys, or using something like AIDE to detect alterations to mirror content without needing reproducible builds. An interesting idea could be certificate transparency style logs that are issued in parallel which list the signatures that a signing key has made such that an attacker would not be able to upload their signatures into the CT log, or if they were, that it would allow notification and detection of the compromise.
A mirror of the source code had alterations.
This would rely on a user downloading and building the kernel from different mirrors. At this point the illusion would already be discovered without reproducible builds as the source tarballs would not match nor would other elements. Again, reproducible builds could find this, but there are also easier ways to detect this in the first place such as signed archives or only using trust mirrors with https.
The developers signing keys were stolen allowing fake builds to be uploaded.
The user would be able to detect the subterfuge by building transmission themself, however as Tavis Ormandy's blog points out "The only way to verify that the untrusted binary is bit-for-bit identical to the binary that would be produced by building the source code, is to produce your own trusted binary first and then compare it. At that point you already have a trusted binary you can use, so what value did reproducible builds provide?"
Additionally the same applies as to CCleaner where key handling improvements would have helped prevent or detect this compromise.
--
Of the 78 listed compromises in this list only 3 (~4%) would be impacted by reproducible builds, and of those 3, all of them have alternate paths to defend and detect compromise that doesn't involve reproducible builds. This reduces reproducible builds to a statistically insignificant counter measure.
When the attacker does attempt to attack a binary artefact it occurs when that attacker has access to the private signing keys allowing them to sign malicious artefacts outside of the build environment.
For this to be relevant to reproducible builds, this would rely on randomised inspection intervals over packaged and signed artefacts. However, there are simpler processes that can be undertaken rather than randomised rebuilds and assertions, such as maintaining a merkle tree of signed packages akin to a certificate transparency log.
To contrast, from the 78 compromises in this document, 30 (~40%) of them were from source code injection earlier in the supply chain, indicating that attackers prefer to compromise source code directly rather than binaries in subsequent build phases.
Conclusion
From the research I have performed I believe that we are unable to prove our hypothesis:
- Reproducible builds would have impacted or prevented historical and current threats against supply chains.
Nor can we reject the null hypothesis:
- Reproducible builds would not prevent historical or current threats against supply chains.
Reproducible builds have a narrow scope of what they can detect and protect, yet it requires a staggering investment of time and resources to achieve. The scope of relevance that reproducbile builds cover is also not a target area for attackers.
In addition, within the scope that reproducible builds do protect, there are alternative measures that can be undertaken to protect these elements of a build process. You may have already noticed things like the offline build hosts which limits the ability for an attacker to compromise these stages of a build in the first place.
Finally, it is important that whenever we discuss a threat, we can not enter the discussion with solutions already in mind. We need to analyse the data, the possible attacks, and then derive solutions that protect from those attacks.
We need to expend effort in places with meaningful outcomes based on risk assessment, rather than propose solutions and force them to stick.
Special Mention
I noticed in the review of each compromise, the Linux Mint ISO compromise.
This attack would actually still be viable today because of how many distributions rely on third party mirrors that operate over http. Most of these distributions still attempt to ask users to perform complex and obtuse GPG verification steps of ISOs. Security must be accessible and usable to all persons, else it's just theatre.