How does it g snap out of a에서도?
Watch we know all these amazing people.
Off your oven, go ahead.
I'm going to go ahead and just start you here, then take your time for our next slide.
It's going to be done by Robyn, and his team is relying on more transparent transport resources for arts-linics packages.
So let's give him our new attention.
Hello everyone.
Thanks for attending this talk.
So yeah, today we're going to talk about the decision that we took an arts-linics to rely on what we qualify as more transparent and trustworthy sources for our packages.
This is basically our response to the supply chain attack that's on Linux distribution, software from recently.
Before we dive into it, allow me to introduce myself.
My name is Robyn.
I'm also known as Antis on the Internet.
I am a Linux system and DevOps engineer by day, and an open source contributor by night.
I am primarily involved in arts-linics in case it wasn't obvious yet.
But I also contribute to reproducible builds and a bit to Alpine Linux as well.
So let's start with a few definition.
First of all, what's the supply chain?
Of course, supply chains are very generic term that can apply to a lot of things.
But I came up with a simple definition for the Linux packaging ecosystem, as this is what we're talking about right now.
So the Linux packaging supply chain is composed of all the elements that are involved in building and delivering packages to users.
Whether it is technical components such as our build script, etc.,
but also the infrastructure the package is built on, the infrastructure that is used to deliver the package to users.
And more importantly, also the individuals included into this whole process.
So what's the supply chain attack then?
Well, as supply chain attack is when one or more of those components is compromised, leading in undesired and unexpected behavior and malicious code being introduced in otherwise trusted packages.
Now do we have do we have protections against supply chain attack?
Yes. Fortunately, we have some mechanism to prevent or at least detect such supply chain attack.
But monitoring and securing the entire supply chain remains a challenge, specifically the very early part of the supply chain.
But we're going to see that later.
So let's go through a quick list of the few of the mechanisms that we have to protect ourselves against supply chain attacks.
First of all, we have checksums.
So when we download the source for a package, we generate like a hash of the source.
And then this hash is locked into our build script as the hash that we expect the next time we're going to download or use the source.
So that ensures that the sources have not been returned in between multiple downloads or rebuilds of the package.
But on much checksum, could indicate a supply chain attack, but it may also have different reasons.
We have a few upstream that have this bad habits of recreating a release, like they say they made a typo in the man page.
And instead of cutting a new point one release, they will discrificly like delete this one and recreated.
Please don't do that, guys.
Or it can be get hub that decides to change their algorithm and get archive that will change every text sum of every get-up total that happened.
They reverted it, but once that happened.
And as I said, this is technically more of a locking mechanism rather than a natural integrity check.
Once we generate the hash, we expect this to never change and we lock the source that way.
But it's not like it guarantees anything in the content of the source in the first place.
Then we have the reproducible builds.
Reproducible builds defines that from the same source with the same build option and the same build environment.
Anyone should be able to reproduce the same artifact bit for bit identical.
Cool thing with the reproducible builds is that it allows to independently verify the path from source to binary code.
Entering that no supply chain attack occurred in between.
So for instance, I has a package maintainer.
I could technically inject the malicious patch into my package before delivery to user.
Of course, all of this is very theoretical, right? I will not do that.
But I technically could. And reproducible builds is a way to verify that I did not abuse my position.
Because you could try to reproduce the package and you should arrive at the same exact result as the package I am delivering to you.
And if you don't, then I may have do something that's thing.
The other cool thing with reproducible builds is that it gives you a technical verification process.
So you don't have to trust me as an individual. You don't have to put your trust in me.
You can actually verify it yourself with a technical process, which is obviously more difficult to compromise than an individual.
While it's always a desired outcome for a distribution, reproducible builds doesn't always cover the whole supply chain though.
So that's a bit of a limit of reproducible builds.
Specifically, the very early part, as I said, that I'm going to demonstrate that in a minute.
And the last mechanism we can go over is cryptographical signatures with like open PGP keys, like new PG signatures.
This hack does an additional integrity check by guaranteeing that an artifact has not been tempered with.
Since it was signed, I think this is an important part here.
So when you sign an artifact, if it gets tempered with, then the signature verification will fail.
So this is an additional integrity check.
And the other cool thing with signatures is that it allows for the authentication of an identity tied to the signer.
So it's basically someone to blame if something that happens.
So that's cool to have someone that is somewhat responsible for whatever they are delivering.
But set identity may or may not relate to an actual person.
Like an artifact could be signed by an identity that relates to the release team, like arsenics release team.
And you don't really know who that is, right?
This is a team.
So it will not always tie to an actual person.
And it may or may not correlate with the creator of the artifact.
The person that signed the artifact is not necessarily the person that created it in the first place.
And there is no way to establish a natural connection between the creator of the artifact and the signature.
We just assume that's the case, but it may not be.
Also I'm not going to spend too much time on this, but realistically signature are only relevant.
If they are accompanied by a trust path, basically if you maintain a chain of trust between the different keys you trust.
If there are multiple keys involved in signing artifact and you just blindly had new keys to your hearing, then you're doing it wrong.
It basically defeats the purpose.
You want to make sure that if you trust that key and a new key is involved, you have a path between the first key you trust and the second one.
Creating the chain of trust that I'm talking about.
If you don't have this bridge, then it defeats the purpose and you might as well just not verify signature, basically.
So all of this to say, we have mechanism and yet you and yet this happened.
So just a quick context here, so XE is a library to compress archive, basically.
And it was back door targeting SSH, which would theoretically allow any untrusted third party to access to your mission remotely.
And this happened, this by the upstream package source being locked behind checksum.
Being cryptographically signed as well.
And this by the infected package being reproducible as well.
So the three mechanism I showed you were correctly applied here and yet the package got back door.
So how did that happen?
Well, there are a few things to note here.
First of all, the malicious code was introduced in the upstream source by one of the upstream maintainer themselves.
Which as I said is a bit difficult for us to verify because such early part of the supply chain are out of our control.
It's very difficult for us as downstream or distributor to have some verification mechanism on the very upstream part.
So this makes it very difficult for us to detect.
And another thing that is important to note here is that part of the malicious code, actually the trigger for it,
was obfuscated by only being present in the custom-made release table.
So what I talk about as custom-made table is those distribution tables that are distributed as an extra artifact of the release,
all the table that you download from upstream website.
And a lot of packages build their packages against that source.
And yeah, it can be abused.
So if you look at the raw sources and the git repository, there was no trace of that trigger for the malicious code.
But if you download that extra table, then the malicious code was there.
So that implies that there are different types of sources that we can use for our packages.
I'm going to name a few, not all of them.
But the first obvious one is plain vcs object. So just check out of the git commit or a git tag,
basically a clone of the repo.
You also have auto-generated table, but git force.
So that is the GitHub auto-generated table or the git lab one or the code one, whatever you use.
And you have those in-famous custom-made release table.
So, oh, well, jump a bit too far here.
Why do we need custom-made release table in the first place? Why do they exist?
Well, the generally easier and most straightforward to work with on the packaging point of view.
Because there are some upstream sources that are based on technologies that require
preliminary step before you're able to work with it on the packaging side.
Now, if there are packages in the room, I may raise some PTSD for you guys,
but stuff like auto tools, hits of modules, translation file, manpages.
So, yeah, auto tools, for instance, needs this very opaque auto-gen bootstrap script stuff that you have to run before you're able to work with sources.
Gits of modules are used to pull third-party dependencies that you may or may not need,
and you would need to do that before you're able to package your sources.
Translation files may need to be generated, manpages as well,
and usually what upstream does is that it gives you this custom-made table,
well, all of these things are already done, so you don't have to take care of that on your side.
There are also historically signed more often than other source type,
probably as a way for upstream to promote this type of sources against other types.
But, there are also very opaque by nature and more difficult and involved to audit,
as compared to a get-repo that is publicly readable.
So, in my opinion, historically we have put way too much trust on signature over other mechanisms.
We'll come back to that later.
And this is, as opposed to, quote, unquote, transparent sources.
So, for us, this is like plain VCS objects or just a check out of a specific git tag or specific commit,
or assuming that you trust those platform, we also decided that,
GitHub, auto-generated table, or Git level, or Git-level, are OK,
given that we trust those platform, but, you know, all our code is hosted on GitLab,
so hopefully we trust GitLab.
And those sources are publicly readable, easily accessible,
and therefore generally more audited,
a way more pair of eyes on a Git repository, rather than on the content of those custom-made
table, right?
It gives you this feeling of what you see is what you get.
So, this is an oversimplified image, but I usually like to see transparent sources as a transparent
box, where I can see the content without having to open that box.
And the custom-made release table would be in the pack box.
And the signature on top of the custom-made table would basically just be a sticker on it
that says, trust me, it's fine, you know?
And I don't want to trust you, actually, I don't want to trust an individual.
So, and you kind of have to do that with custom-made table.
Our transparent sources, I can just look at it and say, OK, this is what I'm going to get.
Now, I've said it may involve some eventual cost in terms of complexity into packaging,
because stuff like auto-gen, fraud at all, gets some modules, et cetera.
Those will be on our side, we will need to do that.
But we now think that it is worth the effort, because distributing packages based on more
transparent sources is generally a good move both for us and for our users.
And as a matter of fact, you probably cannot see it clearly here, but as a matter of fact,
this is the fix that we applied on the XE infected package, and the fix is mostly just to switch
to get sources rather than this custom-made table.
And you can see that we had to, where is it?
Here, we had to have this auto-gen step that we did not have to do beforehand,
thanks to the custom-made table.
Now, we just do it, and that's fine, honestly.
And it's not like a bulletproof source, but at least it is more difficult to
obfuscate malicious stuff in those source.
So, this is what we conclude from that study.
Basically, so there is a link here.
You have a QR code at the end to get the slide, so you can follow the link.
So, the link is basically a package, I switched from custom-made table to get
table, and it implies, like, I don't know, 15 bits of modules depend and see.
So, the package went from this to this, basically, in terms of a build script.
But it's still with the effort, because even if the cost of more complexity,
we now think that those QR-on-code trans-pan sources and more trustworthy sources
should generally be considered a preferred source for our packages.
We've got in cryptographical signatures, so, like, GPG signatures, basically.
We only think that they are really relevant if a company by a trust path.
So, basically, maintaining a chain of trust.
And if you are not able to maintain that chain of trust, you might as well drop key verification.
And if a choice has to be made between the two, so if you have to choose between
trans-pan source or opax source, but signed, we now think that trans-bar sources
are its weight signatures.
Of course, in an ideal well, we would have both, both trans-pan sources and sign sources.
So, we encourage now our package and maintainer to establish communication with upstream,
basically telling us our expectation on that front.
And it should be very easy to do, basically, signing GIT tags is just what we need.
So, we can base our package against your GIT tags and have that signature as well.
Maintaining a trust path for us should be very easy as well.
It could be a signed commit, it could be a signed message.
One thing that upstream usually does is having a maintainer.md file into the repo,
and you would have the key fingerprint of every key that are expected to sign releases.
And just having signed commits, it's time you had a new key or remove a key from there,
is enough for us to have a chain of trust.
So, yeah, that should be an easy step for upstream to do.
And we are now establishing communication with them to hopefully be an ideal world of trans-bar
and sources, and signed sources.
There are a bit extra resources about all of what I just said.
So, you have the XC, Arch sinic security advisory, which says, again, that part of the malicious code
was confiscated into the custom make-to-all bowl, and that the fix was for us to switch to the GIT tags instead.
You have this RC that I can jointly wrote with David Runger, which is another Arch sinic maintainer.
And this is basically what I just said in that talk, but in a very, very long document with a lot of technical things.
So, if you want to get further on this subject, there's a link to that RC.
And you have also what source.org.
That is a website that was created by KPCYRD.
He is an Arch sinic, also an Arch sinic maintainer.
And this website is very cool, because you go to what source.org, and then you tap for instance batch.
And it will show you what sources every major Linux distribution are basing their batch package on.
So, you can see that, for instance, arch is basing its package against the custom make-to-all bowl and fedora does it against the GIT tag, et cetera.
So, that allows you to see what distribution are generally using for a package in terms of source.
A few contacts information, and says arch sinic.org.
If you want to talk to me, I have a website as well. I am on the fediverse.
And QR code to get the slides, if you want to follow the links.
And that's about it.
Thank you.
Any questions for all?
We've got a couple here.
Yes?
Do you prefer a GIT tag over GIT tag?
I mean, a question.
Oh, yeah.
So, do we prefer a GIT tag instead of what do you mean GIT ash?
We have both actually.
Our package tooling is able to generate a a shot to 56 ash for VTS object.
So, we commit a tag, et cetera.
So, we actually have both.
And question?
Yes?
What is help?
This is the sources to oxygen.
do you often tend to be creating the type of what is recruiting civil?
Of course, of course, the question is, would it help if the steps from going from the
raw sources to those custom-made tarble basically, so out of general of that, where reproducible
or at least more transparent?
Of course, that would help.
This is one thing we are trying to do with upstream and discussing with them is to give
us the information on what's actually done from going to this source to the custom-made
tarble, that allows us either to replicate it on our side, so we grab directly the transparent
source, or even better if it is reproducible.
We could technically be based on the custom-made source and with reproducible bills, it
would be fine.
So, that would definitely help us.
Okay.
One comment on that, what that is, it's always, that's the rule, the one to have the source
code in the proper form of modification, which means all the generated files are not
proper form, so we will always want to have the transparent form that have all our way
the generator stuff and generate it on our own again.
I mean, that is obviously a good, oh, I sure will be a good person.
So, Debian does it differently, basically, what they do is that they grab the tarble, remove
the custom stuff and redo the custom stuff itself, basically?
Yes.
So, sort of, that's the expectation.
Okay, that's the expectation.
That's also a lot of work, I'm not sure if we are willing to do this, Debian is a volunteer
based as well, true, but yeah, of course, that would work, but for this to work, we would
also need to be like a policy wise and everyone doing that, which is also a bit difficult
for us to do.
But then we have to have all the tools, we have to have all that does it generation.
True.
What could be good, or as well, is just not using all the tools, that would be even better.
Other questions?
Yes?
Actually, some extreme cases, we have, like, multi language, one way of thought, where to
generate, I will smile, I don't actually need to see many, can we fight and maybe need some
JavaScript?
Would you say, is it always supported, and sometimes, absolutely doesn't want to cooperate,
because it works for them.
That happens, like, as so, question is, there are some upstream build system that are
complex to work around, and sometimes upstream doesn't really want to cooperate with
us.
How do we handle that?
Well, we actually decided to not rely on PiPiTarble for Python packages, mostly because
of the same issue, and some upstream would actually tell you that they only expect PiPiTarble
to be built.
Yes?
And if it's the PiPiTarble created by some data for GitHub automation, is it more
to support you and it's safe to say something?
So, what if the steps to build this custom-tabble are documented or inside a CI that we
can know that?
Of course, that's better, but it's also more work.
Ideally, we should find a good middle ground that would work both for us and for them.
So, yes, that would work, and if we have to do that, we will do it.
This is not about reducing our number of packages, because I'm sure no one's to cooperate.
We want to keep our same repository state that we have, but that would be okay, but
not the most desirable, basically.
Yes?
Is there a systematic way to check up through the change to get the stream to stream?
We now have one.
We did not have one before.
Oh, yeah, repeat the question.
How do we do if upstream recreated attack, basically?
How can we identify that and eventually confront them on that front?
Well, with what source.org, the website I said, web source.org is now registering every
of the commit-ash and commit-ID of a tag.
And if it changes, you can see that way, and it also gather a list of files from the previous
tag and the new one, so we can also compare the list.
So this is where we're usually relying on the what source.org, or if our build service
tail-have-the-old sources, then we can use that as well.
Yes?
Yes.
So, are our packages reproduced by third-party people, or just by us?
Basically, actually both.
So we have a report of our reproducibility of packages.
So we have about 90% of our packages in the report that are reproducible.
And this rebuilder is maintained by us, but on a different machine and the build machine.
Okay, I think that'll be the last question.
And we also have like third-party user that run their rebuilder instance and that also shows
the result.
We can compare our result with also third-party result, and they are usually on-par.
Thanks.
All right.
I think that's it.
Thank you.