All right, can everybody hear me just fine?
Yeah, good.
All right, so hi.
I'm Dimitri.
I'm a PC student in the CA in Paris.
And I'm going to present some work, some current work that I've been working on with Michael
who's just there.
And Stephanie was a curly.
So as I'm sure you may know, there's been a bunch of recent backdoor attacks targeting the supply chain.
So there are two sort of goals here.
Either you compromise the project directly, and you get access to that project,
or you compromise some core project that has a lot of dependence on it.
And through that project, you pollute the entire supply chain.
So the attack factors that we've seen in real life is either people trying to push some malicious commit
as was the case with beef feed people just hijacked some get accounts and managed to push commits.
Or you go the tarble route where you basically replace a legitimate release tarble by an infected one containing a backdoor.
And that's what happened with VSFPD and pro 3D.
And then you have exit utils, which is kind of both, right?
And it's even worse because somebody who is supposed to be a trusted maintainer.
So these were all detected up to a few days after injection, which is already not bad,
but it was largely due to luck and just sheer manual effort, right?
So we need to have something better than this to catch these things.
So how do we detect backdoor's in general?
Well, the historically of doing it is basically manual reverse engineering.
So there's not really a simple algorithm you can use to do it.
You kind of just have to dig through a binary.
And there are tools to help with automating parts of that.
But we talked about this yesterday.
Basically a year ago we introduced a new approach that's based on fuzzing that largely automates this.
So basically the way that approach works is you collect a set of representative inputs.
So those are common inputs to a program.
And for every new input you compare it with the representative inputs.
And behavior is different.
You signal suspicious input because that could be a backdoor.
However, all of these are after the fact detection, right?
So at that point the backdoor is already in the release and the binary is being distributed.
So it's kind of too late.
Like people might have been impacted already.
How do we prevent bugs from ever making into releases?
Well, one way actually is to use fuzzing in CI.
This is already in use.
Just for those of you who don't know fuzzing is a brute force automated test generation approach.
Or you basically spam your target program with a bunch of inputs.
And you're hoping to make it crash, for example.
And that might be a symptom of a vulnerability.
So this already exists for many big oversource projects that you know.
And the reason why is that it's highly automatic.
So it's pretty easy to set up.
And you're going to just let it do this thing.
There's an extremely low fuzzing rate.
So usually it's basically zero.
If you have a crash then that in and of itself is a problem in most cases.
And also it can work with constraint resources.
So even in a CI job in a very short CI job.
You can still get fuzzing to work.
But this can only discover basically crash type vulnerabilities.
So why don't we just apply the approach we talked about yesterday and the security
Deverem in the CI.
Surely you can just plug it in actually no.
So there are a bunch of problems.
So first of all, it's way too slow.
So the approach, if you caught the talk yesterday.
Is targeting binary only programs.
So there's a huge like emulation overhead related to that.
And so you can't just run that in CI.
You can't run in 10 minutes.
And the bigger problem actually is the false positives.
So it's not that the tool produces many false positives.
It's that it produces false positives basically every time.
And this is obviously non-tractable for the CI pipeline.
You can't have your CI telling you there's a back door.
Every other commit when there's none.
So we propose a new approach here.
So we observe that there's a rolling effect in CI, right?
So you have your version n and plus one and plus two and so on.
So instead of just treating them independently as the state of the arts tool to do,
why don't you just use the history basically of the project to get some more information.
So for example, if you assume that the previous version contains no back doors,
then you can do a bunch of stuff.
First of all, you can use its fuzz or generated inputs as a representative of inputs.
And that gives you a bunch more behaviors to compare to.
And then you can go even further and you can say that if I observe something being weird,
quote unquote based on these characteristics in both versions, then it's probably fine.
Because since the previous version is back to our free, like the behavior is the same.
But if I only observe it in the new version, then there's a problem.
So let's go over an example so you get what I mean.
So PHP ships with a built-in HTTP server that actually was attacked via backdoor in 2021.
And we're going to fuzz it with this method to try to detect the backdoor.
So at some point the fuzzer will generate a request to the server with a huge value for content length.
So that's a header.
And basically, as you can imagine the server, we'll just refuse to treat the request.
Like you're shooting me with like a 400 terabyte size request.
I can't treat that so no.
So that leads to different system calls on the server side.
And this would have been saved as suspicious by the existing tools, right?
But if we run it in the previous version of the same HTTP server,
it will react in the same way, right?
This is legitimate behavior actually.
So since both react in the same way, we can safely prune the input away.
And we can remove what would have been a false positive.
So this method allows us to drop the false positive rate by a lot.
And of course, as I said before, speed is still an issue.
Since before we were targeting binaries from firmware, from multiple platforms,
we were largely using Q and U to be able to handle that.
So we were emulating.
This is nice, but it's also extremely slow.
So instead of doing that now, since we're on the developer side,
and we have access to the source code, we'll just use AFL plus pluses,
compiler paths to inject further instrumentation into the source code directly.
So that gives us a much higher throughput for the fuzzard which speeds things up.
So we need to validate this to make sure it works.
So first of all, we need to evaluate whether back to detection has been impacted.
And the way we do that is we test on three real world attacks to which have CVEs attached to them.
And 10 synthetic attacks were basically we created back to recent injected them in popular open source projects.
And then we also want to test false positive filtering.
So we can't possibly test on every commit in the history of the repo.
It's kind of too much.
So what we do is we borrow this approach from another paper actually.
And what we do is we cluster commits and sequences.
So you have sequences of three commits.
And then you compute how many files were changed in a sequence and how many lines.
You take the average of the three commits.
And then you put them in buckets.
So small medium high.
And you take all possible combinations of the buckets.
So that gives you a set of representative commits where you have many files changed.
A few files changed, many lines, a few lines so on and so forth.
We do this twice because many commits actually modify things that do not alter the behavior of the program,
such as documentation, stuff like that.
So one time we just do it indistinguishably throughout the entire history.
The second time we only pick commits that actually alter source code files.
And that gives us four hundred and three to commit pairs to evaluate on throughout the
thirteen different programs that we use.
But actually we can go even further.
So what if we add in another layer of security where if something gets past the project level check,
we can maybe catch it up the district or a little check or the packaging check.
So for example, when a debut in release manager wants to make new release,
they can run the same like ten minute tool on very sensitive packages they say.
And that way there's another net basically that we can cast to catch any factors that might have slipped through.
So in order to test that, we also took the last three debut in a bunch of releases.
And that yields another fifty release pairs where we can test on with much bigger difts this time.
And we also apply the back doors on top of these just to test how the back door detection works in the case of releases.
So grant total 482 version pairs for the commit filtering and 63 for the back doors.
It's more than 500 different version pairs basically to test on.
So the results are that all back doors are still detected as they were with the previous tool.
So we lose nothing on that end.
And actually the detection rate goes up because of the optimizations with the source mode fuzzy.
And this is for ten minute fuzzy conveys.
So this is nothing basically in terms of time.
And false positive wise we have an extremely low rate as well.
So basically only 17 runs out of more than 8,500 had a false positive and had a single one.
And actually we also added a tool that can generate a report showing which lines cause the suspicious behavior or the suspicious system calls.
So you can find yourself in the situation where you have multi thousands of lines.
That gets boiled down to two lines that emit suspicious system calls and as a developer it's much easier to check.
So yeah, in conclusion, back to prevention is possible with an extremely low overhead because you're reusing fuzzy results.
90% detection rate on our benchmark with only 0.2% false positive rates across the entire thing.
Thank you very much.
We have time for a few questions.
Let's go.
So thanks for the call.
We've done a probe to be like included in some repositories.
Our history is like actually history is an absolute nation.
But there is more than institutions at all.
So yeah, the idea is to pitch this to both developers with big open source projects and also destroy maintainers release managers basically.
So you can have multiple layers that check at different intervals.
So a project might check commit by commit, but the release manager might check once every two years or something.
So it's harder to get past both filters at the same time.
And so why are we kind of check for that?
So it's not out yet, but well, if you look at my site or something, I will release it as soon as it's out.
Okay, one more question.
Is language specific or just.
So well, there's a caveat.
So fuzzing, you can do it in any languages.
There are many fuzbers that target many languages.
The fuzbers that we're using for this prototype targets CNC++ programs.
But it's fairly easy to find another fuzzer for your language of choice.
Plug it into our system, our system can take any fuzzer basically.
So it works based on the same premise.
Basically, as soon as you have a system that uses Cisco's, which means basically everything, you can detect back to us this way.
Okay, a few more questions.
Show first the next.
You were saying that you are keeping the fuzzing between one run and the other.
So how much space that?
Because you are kind of trading speed with this space at this point.
Sure.
So the thing is, you don't have to save all the history.
You just have to save the last one.
So sure that might be, I don't know, a few thousand to tens of thousands of test cases.
But those are like small text files basically.
But you only need to save the previous one.
The system does not require a longer history than that.
If you wanted to be really careful, you could save more.
But you can just save the previous one, it will work.
Oh, thank you.
Thank you.
Question of the back end.
Go ahead.
I was wondering if this two or two, like those will help.
Back for designers, maybe more, they don't get worse.
So the benchmark that we use, we can talk about a layer if you want.
It's pretty adversarial.
So we actually, in the paper, we're writing about different attack models against this method.
And from our evaluation, they don't really seem to work reliably.
So you have things that can work a small percentage of the time.
But like, you cannot guarantee that this will get past multiple filters down the line.
Maybe one last question.
Going twice, last question.
No.
Can you go ahead and let you have a look.
So we'll look at the page that you can test with the possibilities.
I guess some sense will still live with the Python context of the data from the input from the parameters.
Are there more?
Sure.
Modern advisors?
Yeah, sorry.
Just like the limit to what you can do.
Sure.
Like, fuzzing blocking constructs, for example, if you put a crypto check like in XU-U-TILS,
a fuzzer can't get past that.
But neither can a symbolic execution engine or any dynamic analysis is kind of like the point of
crypto.
So yeah, you're limited by that.
However, if you do that, you then have a huge blank spot in your coverage.
So you have as a contributor to the project.
You have to explain why what you added cannot be covered.
So for the attacker is complicated.
Thank you.