As the XZ backdoor attack shines a spotlight on the security of open-source software, here are 5 questions to help you gauge the risk.
The XZ Utils attack discovered in March put the open-source software community under a microscope.
This incident is the first supply chain attack on a piece of widely-used open-source software. Open-source software is secure and remains a legitimate option for businesses, but the XZ backdoor raises supply chain management questions: Who is granted authority to make changes? How are these contributors vetted? What security procedures ensure that updates are tested and safe?
The lesson to be learned from XZ Utils is not that open-source software is unsafe. Open-source products can be as safe as managed solutions. The lesson, as it often is with security breaches, is that humans are fallible.
When choosing open-source software for your enterprise, it is good practice to examine the credentials of the team behind the project and query their security protocols.
In the case of the XZ incident, a bad actor took advantage of the maintainer for XZ Utils. Over a period of several years, an attacker using the name Jia Tan worked as a diligent contributor to the XZ library. The maintainer of XZ was a single person, apparently overwhelmed. The attacker was eventually granted commit access and some additional permissions for making releases. This person then added a carefully hidden backdoor to a release which gave an attacker the ability to send hidden commands.
What lessons can we learn from the XZ Utils episode?
I am familiar with the stress and responsibilities of being the maintainer for an open-source project. I founded the FreeRADIUS open-source project and have maintained it for about 25 years, with the help of the staff at Network RADIUS.
Here are a few questions I would ask about any open-source software used by businesses, cloud services, or large organizations like universities.
- Are all commits PGP-signed?
- Are the release files PGP-signed?
- Are generated files committed to the source repository, or is there another method to verify their validity?
- Are releases subject to in-depth tests?
- Is the project maintained by a team or a business?
Using examples from the open-source project, I know best – FreeRADIUS – let me explain how we ensure product security.
All it took to open the door for this backdoor attack was one overwhelmed human. With good security protocols and a team of maintainers, we feel the possibility of a supply-chain attack on FreeRADIUS is very low.
Signed commits and tarballs enhance security
At FreeRADIUS, there are three requirements for anyone to receive commit access for FreeRADIUS:
- be personally verified,
- be trusted,
- have a history of good code.
Only a very small number of people have commit access to the git repository for FreeRADIUS. They are all people I have known for more than a decade and who I have met in person.
All commits by the maintainers are PGP signed. We also PGP-sign the release tarballs. To enhance security, those files are generated internally on secure machines, not on public-facing systems.
Occasionally we merge GitHub PRs from third parties that aren’t PGP signed. Those patches are only allowed if they can be trivially verified to be correct.
Keep it simple: Commit the generated files
At FreeRADIUS, all autoconf-generated files are committed to the source repository.
I recognize that this is largely an ideological choice. This decision was made at the start of the FreeRADIUS project and has not changed since.
Many people believe that generated files should not be tracked in revision control systems such as git. I disagree strongly. The files are small, and tracking them in git allows anyone to trivially check for modified releases, as was done in the XZ attack.
Some open-source projects do not commit these files to revision control, but instead use verified or static Docker images so that users can verify the generated files are correct. We believe that it is simpler for everyone concerned to just commit the generated files to git. We don’t need to create docker images or do anything else to verify that the generated files are correct.
Our generated files are in git, and they are PGP-signed. They cannot be modified by a malicious actor without detection. Anyone looking at the release files can trivially see that the files in the release are the same as the files in git.
Test every release
All releases of FreeRADIUS are run through a large number of in-depth tests. These tests ensure that the code operates correctly and that there are no externally exploitable protocol issues.
We use static analysis tools to find a large number of problems. The git “master” branch also has automated fuzzers that run every night. Any pull requests (PRs) from unknown people have to pass all of the tests before being committed. In many cases, the FreeRADIUS team rewrites the commit ourselves and then closes the PR.
This process helps keep the code clean and safe.
Shared responsibility lightens the load
Since the XZ Utils attack, we have learned the maintainer of XZ was vulnerable to social attacks because a single person was handling this role and they were overloaded with work.
As we all know, at that point, it is tempting to bring new or unknown people on board to help.
In contrast, FreeRADIUS has a business behind it: NetworkRADIUS. The company is on solid footing and has many people involved with FreeRADIUS, and who help share the workload. We have sufficient engineering resources to both maintain FreeRADIUS and help our customers, so the temptation to bring in outside help to maintain the software simply isn’t there.
Embrace the lessons learned from XZ Utils
Now that the tech sector has had time to analyze the XZ supply chain attack, the focus among the open-source community has shifted to lessons learned from this near miss. All it took to open the door for this attack was one overwhelmed human.
Even before the XZ incident, we at NetworkRADIUS aggressively monitored the open-source supply chain for attacks. With good security protocols and a team of maintainers, we feel the possibility of a supply-chain attack on FreeRADIUS is very low.
Business users of open-source software cannot be expected to verify the authenticity and accuracy of each component, utility, and library. But we would suggest that they look for open-source projects which have a solid history, which are maintained by a team or company, and who have strong security policies.
