xz backdoor
The Xz Backdoor Highlights the Vulnerability of Open Source Software—and Its Strengths
Jason Koebler · Mar 30, 2024 at 3:27 PM
The backdoor highlights the politics, governance, and community management of an ecosystem exploited by massive tech companies and largely run by volunteers.
Image: Zulian Firmansyah, Unsplash
Friday afternoon, Andres Freund, a software developer at Microsoft, sent an email to a listserv of open source software developers with the subject line “backdoor in upstream xz/liblzma leading to ssh server compromise.” What Freund had stumbled upon was a malicious backdoor in xz Utils, a compression utility used in many major distributions of Linux, that increasingly seems like it was purposefully put there by a trusted maintainer of the open source project. The “xz backdoor” has quickly become one of the most important and most-discussed vulnerabilities in recent memory.
Ars Technica has a detailed writeup of the technical aspects of the backdoor, which intentionally interfered with SSH encryption, which is a security protocol that allows for secure connections over unsecured networks. The specific technical details are still being debated, but basically, a vulnerability was introduced into a very widely-used utility that chains into a type of encryption that is used by many important internet servers. Luckily, this specific backdoor seems like it was caught before it was introduced into the code of major Linux distributions.
Alex Stamos, the chief trust officer of SentinelOne and a lecturer at Stanford’s Internet Observatory called the discovery of this backdoor “the most interesting hack of the year.”
This is because the mechanism of the attack highlights both the strengths and weaknesses of open source software and the ecosystem under which open source software is developed, and the extent to which the internet and massive tech companies rely on an ecosystem that is largely run by volunteers.
In this case, the vulnerabilities were introduced by a coder who goes by the name Jia Tan (JiaT75 on GitHub) who was a “maintainer” of the xz Utils codebase, meaning they could make commits (update the software’s code) without oversight from others. Critically, Tan has been one of the maintainers of xz Utils for almost two years and also maintains other critical open source projects. This raises the possibility, of course, that they have always been a bad actor and could have been introducing vulnerabilities into earlier versions of xz Utils and other open source projects.
“Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system,” Freund wrote in his initial email.
The open source community is now doing a mix of collaborative damage control, soul searching, and infighting over the backdoor, how to respond to it, and what it means for the broader open source ecosystem. The xz backdoor was seemingly caught before it made its way into major Linux distributions, which hopefully means that there will not be widespread damage caused by the backdoor. But it is, at best, a close call that Freund himself said was essentially “accidentally” discovered.
This is all important because huge parts of the internet and software infrastructure rely on free and open source software that is often maintained by volunteer software developers. This has always been a controversial and complicated state of affairs, because big tech companies take this software, use it in their products, and make a lot of money from them. Many of these open source codebases are maintained by a small number of people doing it on a volunteer basis, and many of these projects have complicated politics about who is allowed to be a maintainer and how a project should be maintained and developed. If a trusted maintainer of a critical open source codebase is actually a malicious hacker, vulnerabilities could be introduced into widely used, critical software and chaos could ensue.
Stamos noted that the backdoor “proves what everybody suspected about the supply-chain risks of OSS. Should hopefully drive some serious investment by the companies that profit from open-source to look for back doors using scalable means.”
The backdoor highlights open source software’s strengths and its weaknesses in that, well, everything is happening in the open.
While a malicious maintainer can commit code that introduces a backdoor, the community can also actively analyze the code and trace exactly what was introduced, when it was introduced, who did it, and what the code does. The project can (and is) rolling back its codebase to an earlier distribution before the vulnerability was introduced. The coding history and email arguments of that user can be traced over time, and the broader developer community can make educated guesses about how this all happened. As I’m writing this, coders are analyzing Jia Tan’s contributions to other projects and the political discussions in listservs that led to them becoming a trusted maintainer in the first place.
On the open source software security listserv, developers are trying to make sense of what happened, and are debating about how and when the discovery of the vulnerability should have been made public (the discovery was made one day before it was distributed to the broader listserv). Tavis Ormandy, a very famous white hat hacker and security researcher who works for Google, wrote on the listserv, “I would have argued for immediately discussing this in the open.”
“We’re delaying everybody else’s ability to react,” he added. Others argued that making the vulnerability known immediately could have incentivized attackers to exploit the bug, or could have allowed others to do so. On Mastodon, software developers are criticizing Microsoft and GitHub for taking down some of the affected code repositories as people are trying to analyze it.
“Hey, it’s totally cool that Microsoft GitHub blocked access to one of the repositories in the very center of the xz backdoor saga,” Michal Woźniak, a white hat hacker who was part of a team that discovered DRM in a Polish train earlier this year wrote on Mastodon. “It’s not like a bunch of people are scrambling to try to make sense of all the right now, or that specific commits got linked to directly from media and blogposts and the like. Cool, cool.” Other coders mused that Copilot, a subscription AI coding assistant created by GitHub, could have integrated some of the malicious code into its training data.
All of this discussion and many of these issues are not normally possible when a vulnerability is discovered in closed source software, which is kept private by the company and whose governance is determined by the companies releasing a product. And that’s what makes all of this so interesting. Not only is vulnerability mitigation being managed in public, but so is the culture, politics, supply chain, and economics that governs this type of critically important software. About the author
Jason is a cofounder of 404 Media. He was previously the editor-in-chief of Motherboard. He loves the Freedom of Information Act and surfing.
More from Jason Koebler
I’ll tell you what it highlights: giant companies like Google, Microsoft and all the others making billions using free software a few dudes maintain for them for free on their own time. Instead of speaking of the vulnerability of open source software, the profiteers should pay them to ensure they have the time and resources to secure their supply chain.
There should be a mandate for companies and profiteers of a library or application to donate x amount of revenue upstream.
For example 1% of your revenue should always go upstream, the next one sends 1% upstream, etc. You can do more of course but imo you should have to do 1%.
I know this is a lot of money in googles example but honestly, its better than just using agpl and keeping them out in the first place. Make them pay their fair share.
My previous employer used to donate to the sole maintainer of a php library we used extensively (I’m not a php developer, so I don’t remember the name). It wasn’t much, but it was something and it is unfortunate that it is not the norm
I fully agree. It should be mandated either by law or at least by license.
It sort of is by license. Not directly but if you’re using one of the more restrictive licenses like GPL 3, it often doesn’t pass legal review due to many of the copy left provisions.
Most companies simply find a similar library that has a more permissive license. A handful will contact the dev and buy a license.
As much as the MIT license has made code more accessible, its permissiveness is the main reason I don’t use it for my own software, unless I really don’t care for it.
Thanks for mentioning this. It was really helpful.
Can you see why I want a more bespoke license which still allows for distribution, change and all that but also asks for you to donate part of your revenue (if you make any, that is) to foss projects?
Because that would streamline the process and would probably find a lot of adopters which would lead to it getting accepted. Probably even more than agpl because you can still make stuff closed source (if we leave the „need to use same license“ out) but you need to pay anyway.
I‘m getting a lot of hate for this btw. People are really unhappy with this idea because for some reason „free“ for them means free beer it seems.
Edit: someone mentioned percentage of employees wages who work on foss projects be factored in which I think is great
I don’t think we need more licenses. OSS license proliferation is bad as it is. IMO, people should do their best to stick with the major licenses: GPL, AGPL, MIT, or Creative Commons if it doesn’t fit the above.
The problem with a tax that you’ve proposed is that it would be nearly impossible to enforce. How would you know which companies are pulling your library?
What I’ve been doing is adding the Commons Clause to my license and that I think helps. I don’t write wildly popular software so I don’t really see people donating or asking to purchase a license.
I personally like the Mozilla model where they donate to various open source projects from a common fund. I’d like to see more stuff like that.
Yeah, the mozilla model seems quite interesting.
After tons of troll messages I‘m now at the point where I will just make everything agpl so nobody can use my stuff if its not the same license and be done with it. I will also make every software I fork agpl if possible which will be a fest.
we should bake something like that in whenever we feel like doing GPLv5 or something.
“free for people, paid for corpos” or something
It’s basically what Redis, ElasticSearch, and others have started doing, but people living in fairytale land are throwing a fit because “iT’s NoT frEe!!11!1”
CC BY-NC-SA 4.0
exactly. I dont understand why FOSS means “go make billions with it, i’ll chew a rock”
Because when projects do it everyone runs away, forks are made, and everyone hates the developers because it’s “not open source anymore”.
I agree with this wholeheartedly,
but if you feel about this methodology strongly you’re going to get hit with nay-sayers that use the same argument anti-VAT people use, as it’s ostensibly the same mechanism: that the developers farthest downstream would have to take the full amount of the percents piled up in their pricing scheme.
Thanks but thats not what I meant. I was talking about a combined 1%. Like, if you used my work, you would need to donate at least (!) 1% of your total revenue to open source projects, ideally evenly distributed. That means the library further upstream would get a tiny amount but not nothing and if everyone did this, the library would have a million or more revenue streams (because libraries are widely used).
So would their salaries for people working on OSS contribute to that 1%?
That could be the case. Thanks for asking and providing valuable new ideas. I think the amount of foss said employees get should factor in, yes.
This wouldn’t work for a few reasons, but the most glaring is that it would incentive re inventing the wheel.
Which is exactly my idea. The AGPL is A LOT worse in this regard since it prevents them from going closed source in the first place iirc. I think many small businesses would gladly use the software and pay 1% of their revenue.
This kind of argument imo is circular because if I build your house for free, you will not build it yourself, plain and simple. If I provide a service, I ought to get paid for it, plain and simple. And if you make money off of my work, you are part of the problem if you dont donate anyway. So making it mandatory imo is absolutely no issue.
Reinventing the wheel is exactly why we should use open source libraries.
Expanding on other unintended outcome here: Different projects have different values. This takes no account for something like Spring vs Apache Commons IO. Or Rails vs nokogiri.
Libraries will be incentivized into breaking apart to maximize revenue.
This isn’t really unlike the unintended consequences of health insurance and how it leads to overpriced services with lots of indecipherable codes for service.
It’s about how the system rewards (pays) for the service. I’m all for supporting open source, but the proposals in this thread are disturbingly anti open source.
I have no idea what you are referring to. Feel free to provide a source.
The consequences of our actions are for the most part completely oblivious until we try it, apart from starting wars and such. But even then its hard to say. So I respect your opinion but I disagree completely. Library maintainers have no reason to maintain libraries because they dont get paid or anything for it, which changes drastically once enough projects use my idea of a license.
The health insurance you are referring to most likely is the american scam version where private companies can suck you dry as they want. Universal healthcare (what happens in some european countries) is what makes going to the doctor dirt cheap or completely cost free. The most disgusting pharma invenstions (like 1000x’ing a cancer medication that used to be dirt cheap iirc) are all american inventions.
Thats the kicker. The system doesnt. They free load. Again, I respect your opinion. My idea is very much open source. It just enforces fairness. Thats all.
Lots of OSS developers are paid by these companies already.
Yeah, and when we find cases like this the best thing for the industry would be for a company or two who are very affected (e.g. Red Hat) to step up and offer a trustworthy person payment for maintaining it.
If they were conpromised, being paid by a company wouldn’t helo. Likely companies would keep it in house, not FOSS. So I think it grows t show how ODD is less open to corruotion. It gets spotted. If this is a state actor that made it haooen, do you really think they have not compromised closed software too? We just have no idea about it not any means to find out.
Company: Here is a security vulnerability in your OSS project, please fix our production is vulnerable.
Random Guy working on OSS library in his free time: Sure, I have some time next month.
Random Guy works full-time, has a family and friends. Random Guy is not your supplier and has no obligations and warranties WHAT SO EVER, even implied. That’s what the license of his project says.
If Company wants it fixed, they better allow him to work full time on it, or pay part time work. Or they pay someone else to maintain Project and send the changes to Project so Random Guy can take a little look and merge if he feels like it. Random Guy won’t just merge company code and be done with it, more code in a codebase needs to be maintained now after all.
This also works with features of course. The time of Random Guy is valuable and if Company wants Random Guy to work on something they use, they’d better pay good money for that time.
deleted by creator
deleted by creator
deleted by creator