Okay. Welcome, everyone. I'm back. I'm Stefan Agarzella. I work in the Redoubt Virtualization
team. And in this talk, I would like to go through all the ways we have in Linux, currently
to do to emulate the Virtio device. This talk was mainly inspired by Thomas, because last
year, I talked a bit about that. In another talk, here at Foss, I'm about Viewstuser.
And Thomas was happy about the comparison between all the way we have to emulate Virtio
devices. So I decided to go a bit deeper and show all the pro and cons we have on all
the way in Linux. So before going into the details, let's start about what is Virtio. Maybe
most of you already know everything about that. But essentially, is an open specification.
The current version is 1.3. There is the 1.4 out for public review. And what they wrote
into the spec is the purpose of Virtio is a straightforward, efficient standard and extensible
mechanism for virtual device. So essentially, they provide, they define a specification
for virtual IU. What is what the spec covers? So essentially, there is a first part is a core,
they define the core component, which each device used to do, for example, feature negotiation
and that kind of things. Then there is, into the spec, for M, a part to define the initialization
phase for the setup. They define all the transport that Virtio device supports. For now,
we have PCI, M, M, M, M, M, M and Tonellaio. And then there is a big section for each
device. So an after block and so on. About the core components, we can say the three main
components are the control path, the data path, and a way to exchange notification. The
control path is mainly used for feature negotiation. So driver and devices essentially agree
on which feature they need to use.
Because for example, we can have a new device with a whole driver
that is not able to use the new feature.
So essentially, they can also vice versa.
So it can be a new driver and the whole device.
So they do this kind of feature negotiation
to agree on what to use.
Then there is a configuration space which
is used for example in an actor card.
It is used to set up the Mac address.
In a V sub device, the device will tell to the guest.
This is your guest, the CID, and that kind of things.
And then the configuration part is
used to set up the data path.
Most of the memory is always allocated by the driver,
by the guest.
So like the data path.
And so the guest needs to tell the driver,
need to tell to the device, say, ER is the word queue.
And so the word queue is essentially
the main part of the data path.
In the picture, I drove a split with queue.
And in the spec, there is also a new version,
which is called part with queue.
About the split, essentially, we have three memory areas.
One is the descriptor table, which mainly contains
pointer to buffers, with the length of the buffer, of course,
flags, that kind of things.
And then we have two rings.
One is the available ring.
And the other one is the used ring.
The available one is used by the driver
to put a century to expose to the device, a new buffer.
So for example, a new pocket.
And the used one, instead, is used by the device
to tell to the driver, hey, I use the buffer.
Now you can take it and clean it or reuse or whatever.
And in the spec with queue, everything is
parked into the descriptor table, mainly for hardware.
Because we will see now we can, or even hardware
are able to expose birthday of devices.
And having three different regions can produce
a lot of memory access, a lot of PCI transactions.
So in order to reduce that contentation,
they define a new format for the VQ, which
everything is parked into the descriptor table.
It's a bit more complicated, but can work much, much better
with hardware.
And about notification, we call kick the notification
from the guest to the host, so from the driver to the device.
And I'll queue, of course, the other way around.
The devices.
We have a lot of devices.
Define it into the spec.
It's an open spec.
So if you have one to define new device,
you can easily open send a serious to define a new one.
The main one, we have a network card,
birthday on that, block device, birthday of block Vsauk,
virtual socket.
The first talker about a lot about Vsauk.
We have a birthday of Vsauk, which support the AF Vsauk
address memory.
We have birthday of FAS for shared folders and others.
And OK, let's go into the topic of the talk.
This is just an overview.
Then we will go in each way.
So the first way, common way to do device emulation
is to do it inside the VMM.
So inside QImo.
This is kind of the standard way.
And also, one of the key points of VMM
is exactly to do device emulation.
And QImo, in our case, provide a lot of birthday devices.
Then we can, in another way, we
have is to move the device emulation
into an external user's process.
The main purpose is for security and reliability.
But as we talk about in Rust VMM,
it's also for flexibility.
We can write in a different language and other reason.
In Linux, for now, we have two way.
We have the Vsauk user, which is the one
I told before in Rust VMM told, and V does, which
is VDPA in user's space.
We will talk a bit more about that later.
Then, another way to emulate the VRTIO devices
are into the host kernel.
This is mainly for performance.
Because we are inside the kernel.
We save a lot of system calls.
And currently, we have two ways.
One is the V host devices.
And the other way is a VDPA software device.
I mean, the VDPA one is a potential one.
For now, we don't have any VDPA software device
implemented into the kernel.
But we did some proof of concept, so it's kind of doable.
And the last one is to move the device
completely into the hardware.
And this was the main reason for VDPA.
Again, for performance, scalability, and that kind of things.
So going back to the first way, the VRTIO device
emulated by the VMM.
As I said, as a commercial scenario, we
can say, Quimu is the factor reference implementation
of VRTIO devices.
You can find a lot of them, not all of them.
But most of them implemented in Quimu.
And why to do that?
Mainly, simplicity.
As I mentioned, Quimu, I mean, a role of a VMM,
one of the key role of VMM is device emulation.
So you have all the API to access the guest memory,
to inject into it, and that kind of things.
And also, portability.
If you write a device in Quimu, essentially,
you can run the device on all operating system
that Quimu supports.
Drawbacks are a real ability to risk.
So your device is in Quimu, a potential bug in your device
can affect Quimu at all, and your VM can go down.
Another is the performance overhead.
So for example, if you have a natural card
and a driver need to send a bucket.
So essentially, the virtual unit driver
will copy a bucket into the VertQ, then send a kick.
So I'll use a VMM exit.
The KVM will write into the IOI event
of D to wake up, Quimu or the IOTrad.
And then there is the device running in Quimu.
We'll access the VertQ, take the bucket,
and maybe send it to a top device.
So we need to do a write.
So again, a system call, go back to the card now.
Send the bucket, then return from the system call
when the bucket is sent, Quimu, the device implementation
will move the descriptor from.
We will put the descriptor into the used ring,
and then need to inject an interrupt.
So again, another system call to write into the RQFD
to tell KVM, hey, inject an interrupt for this device.
So there is a lot of, a lot.
A couple of system calls for each packet
that the device needs to do.
So how to use a VertQ device in Quimu?
This is a simple example, where essentially
we have two block-dab option, which
are part of the Quimu block layer.
And we are telling to the Quimu block layer,
a open the Fedora, QQF2 image, and use
a QQF2 protocol on top.
And then we can link this QQF2 image to a device.
In this case, is a Vertio block PCI device.
But we can use even NVMe, emulation,
or a Vertio scazi, or any other kind of block-device
that Quimu emulates.
So we just need to change the device here.
Vhost.
OK, this was initially introduced exactly
to improve the performance of a Vertio net device.
As I mentioned, when you need to send the packet,
you need to do several SQL.
So what they did, essentially,
is to move the device emulation into the kernel.
The control part is still intercepted by Quimu,
and the kernel will expose a couple of IOCTLs
that the Quimu's mold device can use to essentially
offload to the kernel, the device emulation.
So mainly, where are the Vertios on our access them?
And at this point, and the data part
will completely be handled by the device
into the kernel.
Currently, in Linux, we support three device.
Vhost Net, Vhost Cazian, Vhost Vsoc.
Why, mainly performance, as I mentioned,
having the device into the kernel device, for example,
to talk with that top, don't need to do C-scalls.
So we would say those kinds of C-scalls
are reduced latency and improve the throughput.
But for other use case like Vsoc,
the other reason is being into the kernel.
We are really, it's more easy to interact
with the kernel layers, like the socket layers.
So in the case of Vsoc, we will open,
we will need to open a socket into the kernel,
and having the endpoint of the connection
into the kernel will make us pretty much easier
to instantiate a new socket for AFVsoc.
Drawbacks.
Yeah, of course, we want to go for us.
So we need to pay something.
And there is a security and reliability risk.
Essentially, a bug into the device,
if in Qem, we can crash your VM, ER,
can potentially crash your entire host.
And security again, there is an attack surface
into the host kernel.
So yeah, we need to be careful.
Upgradability, if you want to provide a new feature,
you need to update your host kernel.
And portability.
This is Linux only.
It's only implemented on Linux kernel.
How do you use it?
It really depends on the devices.
For Vsoc is very, very easy.
All the complexity is eaten by Qem.
So you just need to have a Vhost Vsoc device
and specify your guest ID.
Then Qem will set up everything.
We'll create the device for the guest.
And then do the ACTLs to upload the emulation
to the kernel.
Vhost user, I already talked a bit about that in the previous
talk in Rust VMM, but essentially,
the Vhost user was inspired by Vhost.
But instead of having the device into the kernel,
the device now is moving to another process,
another user space process.
So completely separated from the VMM
and from the kernel into the host.
The control part here is a unique domain socket
because we need to move some file descriptor around,
mainly for the shared memory.
So Qem needs to allocate the guest memory in a way
that that memory can be had just by a file descriptor.
And then pass the file descriptor to the device
using the Unix domain socket.
And the data part at this point, at that point,
will be handled completely by the device accessing the guest
memory.
So why to do that?
Yeah, essentially to mitigate the security
and the reliability risk of Vhost.
Because now it runs completely in another user space process.
So if that process can be put in a sandbox, for example,
avoid any escape or can any crash can crash just that process.
We can easily restart a new process again,
doing the emulation.
Another advantage is upgrade ability.
So if you want to update your device,
you just need to run a new process
and potentially just out-plug to your VM.
And flexibility.
We talked about it in the Rust VMM tool.
With the Vhost user, we can essentially
write the device in any other language.
Like we did in Rust VMM, we brought them in Rust
and use it in Qemau pretty easily.
Drawbacks a bit more resources use it.
Because of course, we have other processes.
All devices will run in another process.
So a bit more resource consumption.
And more coordination.
We need to start all the devices first, then start Qemau.
So there is a bit more coordination.
But this could be hidden by management layer,
like libbit, which is used essentially for that.
And portability, I talked about that last year,
was of course, written on Linux, but it's not really Linux
specific, is any POSIC system can run a Vhost user.
Because the only thing we need is, you need to domain
socket and a way to allocate shared memory
address by file descriptor and in POSICs,
we have the SHAM Open C-School that can be used.
How to run it?
OK, it's a bit more complicated.
So in this case, I use it as an example.
The Qemau storage demon, which is an standalone application
that provides exactly the same block layer of Qemau.
But then can expose the device in several ways.
And one way is a Vhost user block.
So this two lines are exactly the same.
I use it in the first example.
So I'm telling to the Qemau storage demon block layer,
which is the same of Qemau.
Hey, open the fedoraQQF2 image.
Use the QQF2 format on top.
And then export this image as a Vhost user block device
on this UNIX domain socket, on the TMP Vhost user socket.
And then we can start Qemau, open the UNIX domain socket,
attaching it to the device, to the Vhost user block PCI device.
One thing we need to use is the memory back end,
a memory back end, which is shareable.
So the memory back end essentially is the way
Qemau allocate the gas memory.
So if you don't use it, if you don't specify the default
is RAM, I think, memory back end RAM, or something like that.
But essentially, Qemau will do just the mall lock,
and which is not OK for the host user.
So we need a shareable memory back end.
And the MMFD is for the default is shareable,
because the memory is essentially
addressed by a file descriptor, and we
can share the file descriptor to the UNIX domain socket.
This is Linux only.
So if you want something portable,
you can use the memory back end SHM
that we recently introduced in Qemau 9.1 last year.
OK, last one, BDPA, which is Vertio
data path acceleration.
In this, BDPA was mainly introduced for hardware,
but we will see we can do also software accelerator
with BDPA.
In the case of BDPA, the data path must
be Vertio compliant.
So the device need to understand the VertQs
and how to elaborate them.
The vendor path, sorry, the control path could
be vendor specific.
So in the case of the hardware device,
we need a little driver into the kernel for the control path.
And why?
Mainly, again, for performance.
We are floating the entire device to the one
example is as Mach Nix.
So this Mach Nix, now some of them
are able to expose Vertio devices.
So we would save several cycles of the CPU to do the emulation,
because the device will be directly
exposed to the guest.
And the VertQs share directly from the guest
to the hardware.
Are you the smart Nix specific hardware physical?
Yeah, physical.
Special driver.
They need to, I mean, some of them
are implemented with R, cheap,
and they need to be able to understand the VertQ.
So the driver essentially will tell
to the smart to the hardware, hey, here is a VertQ.
Please use it for your device to do the emulation.
Very common, or I have to buy it.
It's not very common.
You need to buy, I have the driver, the slide later.
But yeah, it's not really common.
And another advantage is essentially
we support with this.
We support both VM workloads, but even container.
Because VDPA, we have a concept of VDPA
bus into the kernel, where we have the Vhost VDPA bus.
So if your touch device to the Vhost VDPA bus,
the device will be exposed in a similar way
of the Vhost, the second one we show.
So the key menu needs to use Iosity
else to set up the data, but exactly very similar way.
But we have also another bus, which
is the Vertio VDPA bus.
So if you attach your device to that bus,
the driver will be loaded into the kernel, into the host.
So the VertQ will be allocated by the host.
So you can attach your device directly
into your host and use for bare metal application
or containers.
Another advantage is the unified software stack for VDPA.
So as I mentioned, it was introduced for hardware.
But we can write even user space devices.
It's called VDOS, which is similar to Vhost user.
But require some coordination in some models
into the kernel.
And we can also have devices into the kernel itself
like Vhost.
As I mentioned, for now, we don't have
any device like that, but I have an example later about VDOS.
The advantage of VDPA, that all of them,
all of these three ways to implement a VDPA device
are exposing exactly the same way to QEIM.
So QEIM will hardly need to support Vhost VDPA.
And then for QEIM, a point of view doesn't matter
really how the device is implemented.
Drawbacks, portability.
This is Linux only.
So you need things into the Linux kernel.
It's not like the Vhost user, which
is completely agnostic.
So it's really Linux-specific.
And about maturity, it's still working progress.
So we're supported by a few hardware, not a lot.
We support a few.
I mean, for now, I saw just net and block
supported as a birthday of devices.
And of course, the cost.
You need to pay if you want the device
into the hardware, you need to pay for the hardware, of course.
And this is example, again, using QEIM
a storage DMon again, because QEIM a storage DMon
can also expose a VDOS device.
So the first two line again, I exactly the same,
but we are turning to the QEIM a storage DMon.
They expose your devices as a VDOS block.
And with a name, then we can use the VDPA tool
to instantiate really the device.
And at that point, we will find a CSFS entry
that will tell us, hey, this is a VOS VDPA 0.
And so to QEIM, we can just use the VOS VDPA device
PCI with the tractor device where QEIM can do the AocTL
to set up the VrdQs.
And yeah, that's it.
Thank you.
Any question?
The question was about, if we can use a BFF essentially to do
the device emulation into the kernel, can be an idea.
I mean, currently, we don't have any kernel device
VDPA kernel device implemented,
but this could be potential one way to do that.
Yes?
Do the right word to do the A and UF over the A,
if you don't have a different point, we will question
now the VDPA device.
The question was, why not just passing all the PCI device
to the gas to do?
I mean, you can do that.
The main difference is, in this case,
I mean, in that case, your entire PCI device will be
assigned to the gas here.
So you need to support all the PCI stuff.
Here, the device can be emulated, can be implemented
in any way, that only think you need to, of course,
you need to, I mean, your device can implement several
of this virtual function.
And the advantage here is that it
expose a virtual device.
So in your application, you don't need to change anything
into the gas, because the gas is already able to have,
I mean, hopefully I have a virtual network driver,
can just use it.
So you don't need your driver for your PCI device
into the gas.
Yeah, the question was, several years ago,
already were VDPA out, so why it's not so popular now?
I have no idea, honestly.
But I mean, maybe because the hardware vendor
want to lock in, maybe I don't know, or maybe
there are some kind of bottleneck on using that, I don't know.
I mean, I'm not working on VDPA net, but sorry,
we don't have time anymore.
But if you have any question, I will be out.
Thank you.