Re: [PATCH 00/21] Control VQ support in vDPA

[Michael S. Tsirkin]

On Thu, Dec 17, 2020 at 11:30:18AM +0800, Jason Wang wrote:
> 
> On 2020/12/16 下午5:47, Michael S. Tsirkin wrote:
> > On Wed, Dec 16, 2020 at 02:47:57PM +0800, Jason Wang wrote:
> > > Hi All:
> > > 
> > > This series tries to add the support for control virtqueue in vDPA.
> > > 
> > > Control virtqueue is used by networking device for accepting various
> > > commands from the driver. It's a must to support multiqueue and other
> > > configurations.
> > > 
> > > When used by vhost-vDPA bus driver for VM, the control virtqueue
> > > should be shadowed via userspace VMM (Qemu) instead of being assigned
> > > directly to Guest. This is because Qemu needs to know the device state
> > > in order to start and stop device correctly (e.g for Live Migration).
> > > 
> > > This requies to isolate the memory mapping for control virtqueue
> > > presented by vhost-vDPA to prevent guest from accesing it directly.
> > > To achieve this, vDPA introduce two new abstractions:
> > > 
> > > - address space: identified through address space id (ASID) and a set
> > >                   of memory mapping in maintained
> > > - virtqueue group: the minimal set of virtqueues that must share an
> > >                   address space
> > How will this support the pretty common case where control vq
> > is programmed by the kernel through the PF, and others by the VFs?
> 
> 
> In this case, the VF parent need to provide a software control vq and decode
> the command then send them to VF.


But how does that tie to the address space infrastructure?



> 
> > 
> > 
> > I actually thought the way to support it is by exposing
> > something like an "inject buffers" API which sends data to a given VQ.
> > Maybe an ioctl, and maybe down the road uio ring can support batching
> > these ....
> 
> 
> So the virtuqueue allows the request to be processed asynchronously (e.g
> driver may choose to use interrupt for control vq). This means we need to
> support that in uAPI level.

I don't think we need to make it async, just a regular ioctl will do.
In fact no guest uses the asynchronous property.


> And if we manage to do that, it's just another
> type of virtqueue.
> 
> For virtio-vDPA, this also means the extensions for queue processing which
> is a functional duplication.

I don't see why, just send it to the actual control vq :)

> Using what proposed in this series, we don't
> need any changes for kernel virtio drivers.
> 
> What's more important, this series could be used for future features that
> requires DMA isolation between virtqueues:
> 
> - report dirty pages via virtqueue
> - sub function level device slicing


I agree these are nice to have, but I am not sure basic control vq must
be tied to that.

> ...
> 
> Thanks
> 
> 
> > 
> > 
> > > Device needs to advertise the following attributes to vDPA:
> > > 
> > > - the number of address spaces supported in the device
> > > - the number of virtqueue groups supported in the device
> > > - the mappings from a specific virtqueue to its virtqueue groups
> > > 
> > > The mappings from virtqueue to virtqueue groups is fixed and defined
> > > by vDPA device driver. E.g:
> > > 
> > > - For the device that has hardware ASID support, it can simply
> > >    advertise a per virtqueue virtqueue group.
> > > - For the device that does not have hardware ASID support, it can
> > >    simply advertise a single virtqueue group that contains all
> > >    virtqueues. Or if it wants a software emulated control virtqueue, it
> > >    can advertise two virtqueue groups, one is for cvq, another is for
> > >    the rest virtqueues.
> > > 
> > > vDPA also allow to change the association between virtqueue group and
> > > address space. So in the case of control virtqueue, userspace
> > > VMM(Qemu) may use a dedicated address space for the control virtqueue
> > > group to isolate the memory mapping.
> > > 
> > > The vhost/vhost-vDPA is also extend for the userspace to:
> > > 
> > > - query the number of virtqueue groups and address spaces supported by
> > >    the device
> > > - query the virtqueue group for a specific virtqueue
> > > - assocaite a virtqueue group with an address space
> > > - send ASID based IOTLB commands
> > > 
> > > This will help userspace VMM(Qemu) to detect whether the control vq
> > > could be supported and isolate memory mappings of control virtqueue
> > > from the others.
> > > 
> > > To demonstrate the usage, vDPA simulator is extended to support
> > > setting MAC address via a emulated control virtqueue.
> > > 
> > > Please review.
> > > 
> > > Changes since RFC:
> > > 
> > > - tweak vhost uAPI documentation
> > > - switch to use device specific IOTLB really in patch 4
> > > - tweak the commit log
> > > - fix that ASID in vhost is claimed to be 32 actually but 16bit
> > >    actually
> > > - fix use after free when using ASID with IOTLB batching requests
> > > - switch to use Stefano's patch for having separated iov
> > > - remove unused "used_as" variable
> > > - fix the iotlb/asid checking in vhost_vdpa_unmap()
> > > 
> > > Thanks
> > > 
> > > Jason Wang (20):
> > >    vhost: move the backend feature bits to vhost_types.h
> > >    virtio-vdpa: don't set callback if virtio doesn't need it
> > >    vhost-vdpa: passing iotlb to IOMMU mapping helpers
> > >    vhost-vdpa: switch to use vhost-vdpa specific IOTLB
> > >    vdpa: add the missing comment for nvqs in struct vdpa_device
> > >    vdpa: introduce virtqueue groups
> > >    vdpa: multiple address spaces support
> > >    vdpa: introduce config operations for associating ASID to a virtqueue
> > >      group
> > >    vhost_iotlb: split out IOTLB initialization
> > >    vhost: support ASID in IOTLB API
> > >    vhost-vdpa: introduce asid based IOTLB
> > >    vhost-vdpa: introduce uAPI to get the number of virtqueue groups
> > >    vhost-vdpa: introduce uAPI to get the number of address spaces
> > >    vhost-vdpa: uAPI to get virtqueue group id
> > >    vhost-vdpa: introduce uAPI to set group ASID
> > >    vhost-vdpa: support ASID based IOTLB API
> > >    vdpa_sim: advertise VIRTIO_NET_F_MTU
> > >    vdpa_sim: factor out buffer completion logic
> > >    vdpa_sim: filter destination mac address
> > >    vdpasim: control virtqueue support
> > > 
> > > Stefano Garzarella (1):
> > >    vdpa_sim: split vdpasim_virtqueue's iov field in out_iov and in_iov
> > > 
> > >   drivers/vdpa/ifcvf/ifcvf_main.c   |   9 +-
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.c |  11 +-
> > >   drivers/vdpa/vdpa.c               |   8 +-
> > >   drivers/vdpa/vdpa_sim/vdpa_sim.c  | 292 ++++++++++++++++++++++++------
> > >   drivers/vhost/iotlb.c             |  23 ++-
> > >   drivers/vhost/vdpa.c              | 246 ++++++++++++++++++++-----
> > >   drivers/vhost/vhost.c             |  23 ++-
> > >   drivers/vhost/vhost.h             |   4 +-
> > >   drivers/virtio/virtio_vdpa.c      |   2 +-
> > >   include/linux/vdpa.h              |  42 ++++-
> > >   include/linux/vhost_iotlb.h       |   2 +
> > >   include/uapi/linux/vhost.h        |  25 ++-
> > >   include/uapi/linux/vhost_types.h  |  10 +-
> > >   13 files changed, 561 insertions(+), 136 deletions(-)
> > > 
> > > -- 
> > > 2.25.1