Re: [PATCH 4/6] vhost_vdpa: support doorbell mapping via mmap

[Jason Wang]

On 2020/5/29 äå5:16, Mika Penttilà wrote:
> Hi,
>
> On 29.5.2020 11.03, Jason Wang wrote:
> > Currently the doorbell is relayed via eventfd which may have
> > significant overhead because of the cost of vmexits or syscall. This
> > patch introduces mmap() based doorbell mapping which can eliminate the
> > overhead caused by vmexit or syscall.
>
> Just wondering. I know very little about vdpa. But how is such a "sw 
> doorbell" monitored or observed, if no fault or wmexit etc.
> Is there some kind of polling used?


Hi Mika:

It's not a software doorbell. It just allow userspace to map page of 
hardware doorbell directly into userspace.

Without this, for KVM, it needs to trap the MMIO access of the guest and 
write to eventfd, for other userspace driver, it needs to write to 
eventfd. vhost-vDPA's eventfd wakeup function may let the driver to do 
touch the doorbell.

With this, since the doorbell page is mapped into userspace address 
space, guest or other userspace driver may write directly to the 
hardware doorbell register.

Thanks


> > To ease the userspace modeling of the doorbell layout (usually
> > virtio-pci), this patch starts from a doorbell per page
> > model. Vhost-vdpa only support the hardware doorbell that sit at the
> > boundary of a page and does not share the page with other registers.
> >
> > Doorbell of each virtqueue must be mapped separately, pgoff is the
> > index of the virtqueue. This allows userspace to map a subset of the
> > doorbell which may be useful for the implementation of software
> > assisted virtqueue (control vq) in the future.
> >
> > Signed-off-by: Jason Wang <jasowang@xxxxxxxxxx>
> > ---
> > Â drivers/vhost/vdpa.c | 59 ++++++++++++++++++++++++++++++++++++++++++++
> > Â 1 file changed, 59 insertions(+)
> >
> > diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> > index 6ff72289f488..bbe23cea139a 100644
> > --- a/drivers/vhost/vdpa.c
> > +++ b/drivers/vhost/vdpa.c
> > @@ -15,6 +15,7 @@
> > Â #include <linux/module.h>
> > Â #include <linux/cdev.h>
> > Â #include <linux/device.h>
> > +#include <linux/mm.h>
> > Â #include <linux/iommu.h>
> > Â #include <linux/uuid.h>
> > Â #include <linux/vdpa.h>
> > @@ -741,12 +742,70 @@ static int vhost_vdpa_release(struct inode 
> > *inode, struct file *filep)
> > ÂÂÂÂÂ return 0;
> > Â }
> > Â +static vm_fault_t vhost_vdpa_fault(struct vm_fault *vmf)
> > +{
> > +ÂÂÂ struct vhost_vdpa *v = vmf->vma->vm_file->private_data;
> > +ÂÂÂ struct vdpa_device *vdpa = v->vdpa;
> > +ÂÂÂ const struct vdpa_config_ops *ops = vdpa->config;
> > +ÂÂÂ struct vdpa_notification_area notify;
> > +ÂÂÂ struct vm_area_struct *vma = vmf->vma;
> > +ÂÂÂ u16 index = vma->vm_pgoff;
> > +
> > +ÂÂÂ notify = ops->get_vq_notification(vdpa, index);
> > +
> > +ÂÂÂ vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> > +ÂÂÂ if (remap_pfn_range(vma, vmf->address & PAGE_MASK,
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ notify.addr >> PAGE_SHIFT, PAGE_SIZE,
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ vma->vm_page_prot))
> > +ÂÂÂÂÂÂÂ return VM_FAULT_SIGBUS;
> > +
> > +ÂÂÂ return VM_FAULT_NOPAGE;
> > +}
> > +
> > +static const struct vm_operations_struct vhost_vdpa_vm_ops = {
> > +ÂÂÂ .fault = vhost_vdpa_fault,
> > +};
> > +
> > +static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct 
> > *vma)
> > +{
> > +ÂÂÂ struct vhost_vdpa *v = vma->vm_file->private_data;
> > +ÂÂÂ struct vdpa_device *vdpa = v->vdpa;
> > +ÂÂÂ const struct vdpa_config_ops *ops = vdpa->config;
> > +ÂÂÂ struct vdpa_notification_area notify;
> > +ÂÂÂ int index = vma->vm_pgoff;
> > +
> > +ÂÂÂ if (vma->vm_end - vma->vm_start != PAGE_SIZE)
> > +ÂÂÂÂÂÂÂ return -EINVAL;
> > +ÂÂÂ if ((vma->vm_flags & VM_SHARED) == 0)
> > +ÂÂÂÂÂÂÂ return -EINVAL;
> > +ÂÂÂ if (vma->vm_flags & VM_READ)
> > +ÂÂÂÂÂÂÂ return -EINVAL;
> > +ÂÂÂ if (index > 65535)
> > +ÂÂÂÂÂÂÂ return -EINVAL;
> > +ÂÂÂ if (!ops->get_vq_notification)
> > +ÂÂÂÂÂÂÂ return -ENOTSUPP;
> > +
> > +ÂÂÂ /* To be safe and easily modelled by userspace, We only
> > +ÂÂÂÂ * support the doorbell which sits on the page boundary and
> > +ÂÂÂÂ * does not share the page with other registers.
> > +ÂÂÂÂ */
> > +ÂÂÂ notify = ops->get_vq_notification(vdpa, index);
> > +ÂÂÂ if (notify.addr & (PAGE_SIZE - 1))
> > +ÂÂÂÂÂÂÂ return -EINVAL;
> > +ÂÂÂ if (vma->vm_end - vma->vm_start != notify.size)
> > +ÂÂÂÂÂÂÂ return -ENOTSUPP;
> > +
> > +ÂÂÂ vma->vm_ops = &vhost_vdpa_vm_ops;
> > +ÂÂÂ return 0;
> > +}
> > +
> > Â static const struct file_operations vhost_vdpa_fops = {
> > ÂÂÂÂÂ .ownerÂÂÂÂÂÂÂ = THIS_MODULE,
> > ÂÂÂÂÂ .openÂÂÂÂÂÂÂ = vhost_vdpa_open,
> > ÂÂÂÂÂ .releaseÂÂÂ = vhost_vdpa_release,
> > ÂÂÂÂÂ .write_iterÂÂÂ = vhost_vdpa_chr_write_iter,
> > ÂÂÂÂÂ .unlocked_ioctlÂÂÂ = vhost_vdpa_unlocked_ioctl,
> > +ÂÂÂ .mmapÂÂÂÂÂÂÂ = vhost_vdpa_mmap,
> > ÂÂÂÂÂ .compat_ioctlÂÂÂ = compat_ptr_ioctl,
> > Â };