Re: [PATCH 4/6] vhost_vdpa: support doorbell mapping via mmap

From: Jason Wang
Date: Fri May 29 2020 - 05:24:55 EST

On 2020/5/29 äå5:16, Mika Penttilà wrote:

On 29.5.2020 11.03, Jason Wang wrote:
Currently the doorbell is relayed via eventfd which may have
significant overhead because of the cost of vmexits or syscall. This
patch introduces mmap() based doorbell mapping which can eliminate the
overhead caused by vmexit or syscall.

Just wondering. I know very little about vdpa. But how is such a "sw doorbell" monitored or observed, if no fault or wmexit etc.
Is there some kind of polling used?

Hi Mika:

It's not a software doorbell. It just allow userspace to map page of hardware doorbell directly into userspace.

Without this, for KVM, it needs to trap the MMIO access of the guest and write to eventfd, for other userspace driver, it needs to write to eventfd. vhost-vDPA's eventfd wakeup function may let the driver to do touch the doorbell.

With this, since the doorbell page is mapped into userspace address space, guest or other userspace driver may write directly to the hardware doorbell register.


To ease the userspace modeling of the doorbell layout (usually
virtio-pci), this patch starts from a doorbell per page
model. Vhost-vdpa only support the hardware doorbell that sit at the
boundary of a page and does not share the page with other registers.

Doorbell of each virtqueue must be mapped separately, pgoff is the
index of the virtqueue. This allows userspace to map a subset of the
doorbell which may be useful for the implementation of software
assisted virtqueue (control vq) in the future.

Signed-off-by: Jason Wang <jasowang@xxxxxxxxxx>
 drivers/vhost/vdpa.c | 59 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 6ff72289f488..bbe23cea139a 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -15,6 +15,7 @@
 #include <linux/module.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
+#include <linux/mm.h>
 #include <linux/iommu.h>
 #include <linux/uuid.h>
 #include <linux/vdpa.h>
@@ -741,12 +742,70 @@ static int vhost_vdpa_release(struct inode *inode, struct file *filep)
ÂÂÂÂÂ return 0;
 +static vm_fault_t vhost_vdpa_fault(struct vm_fault *vmf)
+ÂÂÂ struct vhost_vdpa *v = vmf->vma->vm_file->private_data;
+ÂÂÂ struct vdpa_device *vdpa = v->vdpa;
+ÂÂÂ const struct vdpa_config_ops *ops = vdpa->config;
+ÂÂÂ struct vdpa_notification_area notify;
+ÂÂÂ struct vm_area_struct *vma = vmf->vma;
+ÂÂÂ u16 index = vma->vm_pgoff;
+ÂÂÂ notify = ops->get_vq_notification(vdpa, index);
+ÂÂÂ vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+ÂÂÂ if (remap_pfn_range(vma, vmf->address & PAGE_MASK,
+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ vma->vm_page_prot))
+static const struct vm_operations_struct vhost_vdpa_vm_ops = {
+ÂÂÂ .fault = vhost_vdpa_fault,
+static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
+ÂÂÂ struct vhost_vdpa *v = vma->vm_file->private_data;
+ÂÂÂ struct vdpa_device *vdpa = v->vdpa;
+ÂÂÂ const struct vdpa_config_ops *ops = vdpa->config;
+ÂÂÂ struct vdpa_notification_area notify;
+ÂÂÂ int index = vma->vm_pgoff;
+ÂÂÂ if (vma->vm_end - vma->vm_start != PAGE_SIZE)
+ÂÂÂ if ((vma->vm_flags & VM_SHARED) == 0)
+ÂÂÂ if (vma->vm_flags & VM_READ)
+ÂÂÂ if (index > 65535)
+ÂÂÂ if (!ops->get_vq_notification)
+ÂÂÂ /* To be safe and easily modelled by userspace, We only
+ÂÂÂÂ * support the doorbell which sits on the page boundary and
+ÂÂÂÂ * does not share the page with other registers.
+ÂÂÂÂ */
+ÂÂÂ notify = ops->get_vq_notification(vdpa, index);
+ÂÂÂ if (notify.addr & (PAGE_SIZE - 1))
+ÂÂÂ if (vma->vm_end - vma->vm_start != notify.size)
+ÂÂÂ vma->vm_ops = &vhost_vdpa_vm_ops;
+ÂÂÂ return 0;
 static const struct file_operations vhost_vdpa_fops = {
ÂÂÂÂÂ .openÂÂÂÂÂÂÂ = vhost_vdpa_open,
ÂÂÂÂÂ .releaseÂÂÂ = vhost_vdpa_release,
ÂÂÂÂÂ .write_iterÂÂÂ = vhost_vdpa_chr_write_iter,
ÂÂÂÂÂ .unlocked_ioctlÂÂÂ = vhost_vdpa_unlocked_ioctl,
+ÂÂÂ .mmapÂÂÂÂÂÂÂ = vhost_vdpa_mmap,
ÂÂÂÂÂ .compat_ioctlÂÂÂ = compat_ptr_ioctl,