Hello everyone, so this is a bit of a mouthful of title, so basically the name of the software
project, what you should remember is a CVMFS. Before we get into the content, we'd like
to suggest you visit our both session, and the open source at 7 stand at K level 1, and
now for the George Will. Is this all right? Perfect. Okay, so I think I need to do
this first here, right here. Okay, so hello, I'm George, Europe, Europe, Europe,
Europe, Europe, Europe, Europe, Europe, Europe, Europe, Europe, Europe, Europe, and I'm a certain fellow
working in the APSFD group since 2025, and especially in the CVMFS project, and to
tell you, like I'll give a brief introduction to CVMFS, and I'll talk to you how can we benefit
from CVMFS on content rights workflows. So something that I was not actually aware of before
landing at turn is the amount of data that that experiments in high energy physics and
practical physics are producing, just the Sun Data Center is starting more than 30
petabytes of data per year, and it's dedicated software to process them, and of course this
software cannot operate only locally, it has to be distributed all over in libraries, over over
the world, and this is the main mission that CVMFS came for. I'll just use a small time
split here to mention easy, I guess most of you should have already heard about them,
it's the European Environment Force certificate software installation was mission is to provide
the uniform experience for people that are working with scientific software to have
a transparent experience regardless of the underlying system they are working on. So probably
you're familiar with the stock, CVMFS is in the file system layer of this stock, actually they
have trust pretty good presentations, for more in the HPC development totally recommend attending
there. So CVMFS, what is it? It's read-only file system in the user space, and I think a good
metaphor to think about it, it is like an on-demand streaming service, but for scientific software.
So software in CVMFS is an organized group in a repository that in principle do not leave
on the user's machine, and CVMFS from the user's perspective is providing a normal
process, browsing an accessing experience to those files. So we can do like the listing or
data accessing, and what's the take home message here is that these those files are not present
on the users' machine, they live elsewhere, and they're being downloaded on demand.
But how does this work? So CVMFS, I think, who is using, under the who is using fused,
and he just comes with the kernel module, and a user space library. So every file system
access is forwarded through VFS to the kernel module, and then in the operating system,
and CVMFS is implementing all the necessary read-only system calls to enable the user
access files. So in principle, if the file is present in the local storage, it will be used directly,
if not, it will be fetched from the network. Okay, but how did that become so available to the
user? So we start from a normal read-ride file system, and then we follow the process called
compublishing, and what's happening under the hood is compression, chunking, and we use
merciletries to sort our data in a content addressable format, which means that basically in
or to access our data, we don't use the path or their name, we just use their content,
and this allows for deduplication, which practically means that the same file or chunk
is never stored wise, which saves a lot of logical space. We have an overview of the order
of magnitude of that later on talking about the unpopular repo.
Of course, okay, then once we have the data in a content addressable format on our server,
then they become accessible from the user's client just by using HTTP, and I guess it's pretty intuitive
that uses a storage is not limited, so we use an LRU clean up policy to respect the available space.
So some numbers here, this is the evolution of the four principal repositories. The
turn is hosting with similar 7FS, plus the unpacked.sern.ca 3PO that we'll talk about later.
At the moment, turn is distributing over two beta bytes of data using 7FS,
more than 8,000 container images and around 290 repositories, and something that came
along the way is that besides being a very good tool for software distribution,
7FS has also it ought to be nice tool for software preservation.
Okay, so a small parenthesis here to talk about container images, and then so
how 7FS is used to facilitate a container image distribution.
So in principle, container images are file system layers that are stored, stored and unindexed
file archive, and this practically means that even if we need a specific
time portion of the archive, we cannot access it directly, we need to have access
in entirety, decompress it, and then access it, usual file system, usual container images in
the physics sphere is around in the order of magnitude of gigabytes, and we end up using a tiny
fraction of it. So we'd like to have a way to be able to access only the essential parts.
So this is already been there. Container D allows for providing some apps, some apps,
other plugins could exactly that we do lazy pulling, and we are only pulling the essential
parts of the archive. There are many alternatives, there's strategy, it's strategy,
so basically, it's one introduced a different way to form a to store container images.
Also, CVMFS comes with dedicated to the synapse author, and it provides a way to publish container
images in an unpacked format, which is also compressed and de-duplicated. It works with Docker
app owner and podman, and also some overly here of the main repository uttering that is providing
an unpacked container images is unpacked attorney.ca, you can see that it's broadly adapted from
2022 until now, if it's not visible well. But we have recently known of 12 terabytes of
compressed and de-duplicated images more than 120 million files, which would correspond to more
than a petabyte of logical data if they were not de-duplicated. So I'll pass this stage here to
underely who has some nice numbers. 100x the compression rate you want, just be honest, that's cool.
So this is a practical demonstration of lazy pooling of container images, which George has just
introduced. So this is all it takes on a fresh CVMFS install to actually use this unpacked
CRMCH repository. You shouldn't be using it in production, but you can do that. So basically we
issue podman command. It's just an example that you can use it with podman. Nothing special with
podman to do that. So you just give it some temp-dir to mount it through the first, then use
specify this CVMFS mount point location, which includes the path to image directory, and then you
give it a comment. So here we just load a particular module with a python. And it took us 25 seconds
and in our cache, we have a cache in the end there is 600 megabytes, but the whole image is over one
gigabyte, so we saved downloading 400 megabytes. So this is just normal pooling.
So even on a very fast gigabyte of link connection, it took over a minute to download it. And in fact,
it can end up like this. So this is also a part of motivation behind lazy pooling to avoid
situations like this to have to go and clean up expeditions or resize your machines with that.
There is a great, so there is exists the documentation for lazy pooling of containers
with the CVMFS, upstream edge check the links in slides. Cubecon 25 had a great talk by
Valentin. So a slides and video of that are available. So here we want to introduce
another new set of tools, modeled after popular five utilities, which offer the convenience of
direct effect on a CVMFS repo. So it can be seen as easier to use tools. They might
provide speed-ups in some modes. They can help avoid running out of disk space in a scratch area
when you are updating CVMFS repo content. So this will become clearer in a bit. So the usual way
to change the contents involves the three stages. So you give it a command CVMFS server transaction.
You make changes in a particular directory and then you issue the command CVMFS server
publish and then the changes after that would become available to clients who mount your repository.
So this is how it looks.
So usually you have mount read-only of CVMFS and then with a transaction command CVMFS mount
stays read-only but your overlay is amounted read-write so you then can jump into this directory
and do your changes. So overlayFS then put them. So they appear in your this directory but
in fact CVMFS itself still doesn't have this content but this directory handled by overlay
FS is called scratch-dure. It holds this temporarily. So then we issue this command and then
it is uploaded into the primary storage. So here we add four gigabyte file to S3 back to the
first repository. S3 doesn't necessarily mean Amazon many, many implementations work.
So these are the three stages we've just seen.
And now is the new tool CVMFS R sync which works slightly differently. So we uploaded it
so we issued the command and then it's there and it never goes into the scratch area.
So how that works? So it uploads files to S3 directly. It pushes metadata changes to CVMFS gateway
host. This is separate service which might which might be present in simple and this in simple
setups it might be not in use at all. So and then it waits for new CVMFS revision to become
visible from the client standpoint. So speaking of S3 implementations so many protocol implementations
work. I know of Seth works. I checked garage works. Minio Azure is a red it works but with a quick.
So again, come on Yed.
It was this visible, okay.
So Stephen Fespoise explores at the moment
a ceiling a pool request, but you know where it is now.
So time for questions, and again don't forget
these two other related events if you want to know more.
So questions?
Quite okay.
Yes.
The impact of 10.48 is quite big, even with all the
duplication and compression.
Does it go from home and interact with management
or also, you know,
get the question?
Yes.
So the size of unpack this quite big,
and if you have troubles with its management,
sort answer is yes.
There is some technicalities.
The implementation specific details that we can improve somehow
to make it actually is,
okay, maybe we don't have the time
calculated into this, how catalogs work,
but maybe want to do the unpacking in the catalog level
and we hope that this will just
easy a lot of the publication process of unpacking
legislation.
So when you demonstrate to me
in the beginning, you talked about a lot of data
to do this.
But then we really switch over to just the software
set.
Do you provide, at least that's what we're talking about
from the software?
You also provide a storage
like data sets, and so on.
Yes.
So there's the,
it was primarily designed and just for
software distribution.
But then there is the possibility to distribute
static data.
And also you can, you can do
sophisticated data that have already been stored elsewhere.
So the idea is there's the external,
external data distribution.
The catalogs are generated in a repository
the same way it would have been done
like publishing software.
And then it can be accessed by the client
directly where they live.
And I think but for not in the content
addressable format anymore.
So if we go via the local process of publishing,
it will be the normal one.
It will be content addressable if not
it would be just accessed the same way.
But through the catalogs provided by the normal publishing.
Okay.
Yes.
Fine.
Have this fuel on facts.
Container of geese and direct communities.
How unpacked,
transient based container images
interact with Kubernetes.
You can use this.
You can launch containers with these as long as you
have a 7 of us mount, which I.
I have no experience with Kubernetes.
If you can set up a Kubernetes host or whatever
that can launch the containers using this command
then yes.
Otherwise, no.
There's a plugin that you can use.
Yeah.
It's the container.
You snapshot that information.
Yes, it's the plugin that does this.
It's the snapshoter that I mentioned just before.
I'm just very moving.
I have this very moving.
I have two questions.
Okay.
Removing files.
I have this removing files in garbage collection
works.
So the reason why the process of publishing
there is a similar procedure to update
the repositories on the server that they.
They leave the exist.
There is a chain of like before the data become available
to the client.
There is a series of.
Proxies and then mirror servers and then there is one
principle server that is responsible for publishing.
And it's there where the updating happens.
For garbage collection, I'm not.
I'm not the guy to respond.
So maybe Valentin.
Yeah.
But the internal.
I cannot say more.
Yeah.
When the other questions,
you see if it has been around for a long time
over a decade.
But if you like it's getting more popular
on recent years.
Is it an explanation for this?
Is it a container?
Support?
Or is it one of the things going on?
Well, it's.
It's.
Not.
But, kind of.
Not.
But, you know, that would be.
And that is.
So, it would be.
So, it's.
Okay.
I.
So, it's.
So.
But not very clear, is the question, does it compress the files?
Is it a lot of work, does it require high, like, beefy machines to be doing, to set up a server?
No, it works on small virtual machines, it on laptop, it definitely works as a container.
So basically, as soon as it's processed, published, it's just serving as HTTP requests from now on.
And it's externalized somewhat, so you can serve HTTP requests with a page or a stream implementation.
And the point to actually scaling up to serve a lot of clients is to add layers of proxying,
and then charging the process and adding more layers and more sophistication to proxying, that's the way you serve a lot.
And the central server doesn't need to be beefy.
I also think it depends on the amount of data that is to be published.
I think maybe if you talk to Serenity, who has a different opinion about it.
So yeah, I mean, for small scale experiments, it may be easy, but if you want to publish a big amount of data,
I think publication becomes an issue, actually there are developments, to ease the publication of sub parts of a directory in different machines,
to make this easier via dedicated entity, which is a gateway.
So my workflow would be it depends on the scale of what you want to publish.
So we have five more minutes for questions, don't be shy.
No questions.
All right, thank you, everyone.
Thank you.