Re: [RFC PATCH 00/22] Enhance VHOST to enable SoC-to-SoC communication

From: Mathieu Poirier
Date: Wed Jul 15 2020 - 13:15:21 EST

Next message: Pawan Gupta: "Re: [PATCH] x86/bugs/multihit: Fix mitigation reporting when KVM is not in use"
Previous message: Matthew Wilcox: "Re: strace of io_uring events?"
In reply to: Jason Wang: "Re: [RFC PATCH 00/22] Enhance VHOST to enable SoC-to-SoC communication"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hey Kishon,

On Wed, Jul 08, 2020 at 06:43:45PM +0530, Kishon Vijay Abraham I wrote:
> Hi Jason,
>
> On 7/8/2020 4:52 PM, Jason Wang wrote:
> >
> > On 2020/7/7 äå10:45, Kishon Vijay Abraham I wrote:
> >> Hi Jason,
> >>
> >> On 7/7/2020 3:17 PM, Jason Wang wrote:
> >>> On 2020/7/6 äå5:32, Kishon Vijay Abraham I wrote:
> >>>> Hi Jason,
> >>>>
> >>>> On 7/3/2020 12:46 PM, Jason Wang wrote:
> >>>>> On 2020/7/2 äå9:35, Kishon Vijay Abraham I wrote:
> >>>>>> Hi Jason,
> >>>>>>
> >>>>>> On 7/2/2020 3:40 PM, Jason Wang wrote:
> >>>>>>> On 2020/7/2 äå5:51, Michael S. Tsirkin wrote:
> >>>>>>>> On Thu, Jul 02, 2020 at 01:51:21PM +0530, Kishon Vijay Abraham I wrote:
> >>>>>>>>> This series enhances Linux Vhost support to enable SoC-to-SoC
> >>>>>>>>> communication over MMIO. This series enables rpmsg communication between
> >>>>>>>>> two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2
> >>>>>>>>>
> >>>>>>>>> 1) Modify vhost to use standard Linux driver model
> >>>>>>>>> 2) Add support in vring to access virtqueue over MMIO
> >>>>>>>>> 3) Add vhost client driver for rpmsg
> >>>>>>>>> 4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for
> >>>>>>>>> ÂÂÂÂÂÂ rpmsg communication between two SoCs connected to each other
> >>>>>>>>> 5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication
> >>>>>>>>> ÂÂÂÂÂÂ between two SoCs connected via NTB
> >>>>>>>>> 6) Add configfs to configure the components
> >>>>>>>>>
> >>>>>>>>> UseCase1 :
> >>>>>>>>>
> >>>>>>>>> ÂÂÂÂ VHOST RPMSGÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VIRTIO RPMSG
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂ +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> +-----v------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------v-------+
> >>>>>>>>> |ÂÂ LinuxÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ LinuxÂÂÂ |
> >>>>>>>>> |Â EndpointÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | Root Complex |
> >>>>>>>>> |ÂÂÂÂÂÂÂÂÂÂÂ <----------------->ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> |ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> |ÂÂÂ SOC1ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ SOC2ÂÂÂÂ |
> >>>>>>>>> +------------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +--------------+
> >>>>>>>>>
> >>>>>>>>> UseCase 2:
> >>>>>>>>>
> >>>>>>>>> ÂÂÂÂÂÂÂÂ VHOST RPMSGÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VIRTIO RPMSG
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂ +------v------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------v------+
> >>>>>>>>> ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂ |ÂÂÂ HOST1ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ HOST2ÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂ +------^------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------^------+
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> +---------------------------------------------------------------------+
> >>>>>>>>> |Â +------v------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------v------+Â |
> >>>>>>>>> |Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
> >>>>>>>>> |Â |ÂÂÂÂ EPÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ EPÂÂÂÂÂ |Â |
> >>>>>>>>> |Â | CONTROLLER1 |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | CONTROLLER2 |Â |
> >>>>>>>>> |Â |ÂÂÂÂÂÂÂÂÂÂÂÂ <----------------------------------->ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
> >>>>>>>>> |Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
> >>>>>>>>> |Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
> >>>>>>>>> |Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â SoC With Multiple EP InstancesÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
> >>>>>>>>> |Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â (Configured using NTB Function)Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
> >>>>>>>>> |Â +-------------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +-------------+Â |
> >>>>>>>>> +---------------------------------------------------------------------+
> >>>>>>>>>
> >>>>>>>>> Software Layering:
> >>>>>>>>>
> >>>>>>>>> The high-level SW layering should look something like below. This series
> >>>>>>>>> adds support only for RPMSG VHOST, however something similar should be
> >>>>>>>>> done for net and scsi. With that any vhost device (PCI, NTB, Platform
> >>>>>>>>> device, user) can use any of the vhost client driver.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ÂÂÂÂÂÂÂ +----------------+Â +-----------+Â +------------+Â +----------+
> >>>>>>>>> ÂÂÂÂÂÂÂ |Â RPMSG VHOSTÂÂ |Â | NET VHOST |Â | SCSI VHOST |Â |ÂÂÂ XÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂ +-------^--------+Â +-----^-----+Â +-----^------+Â +----^-----+
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> +-----------v-----------------v--------------v--------------v----------+
> >>>>>>>>> |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VHOST COREÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> +--------^---------------^--------------------^------------------^-----+
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>>>>>>>> +--------v-------+Â +----v------+Â +----------v----------+Â +----v-----+
> >>>>>>>>> |Â PCI EPF VHOST |Â | NTB VHOST |Â |PLATFORM DEVICE VHOST|Â |ÂÂÂ XÂÂÂÂ |
> >>>>>>>>> +----------------+Â +-----------+Â +---------------------+Â +----------+
> >>>>>>>>>
> >>>>>>>>> This was initially proposed here [1]
> >>>>>>>>>
> >>>>>>>>> [1] ->
> >>>>>>>>> https://lore.kernel.org/r/2cf00ec4-1ed6-f66e-6897-006d1a5b6390@xxxxxx
> >>>>>>>> I find this very interesting. A huge patchset so will take a bit
> >>>>>>>> to review, but I certainly plan to do that. Thanks!
> >>>>>>> Yes, it would be better if there's a git branch for us to have a look.
> >>>>>> I've pushed the branch
> >>>>>> https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc
> >>>>> Thanks
> >>>>>
> >>>>>
> >>>>>>> Btw, I'm not sure I get the big picture, but I vaguely feel some of the
> >>>>>>> work is
> >>>>>>> duplicated with vDPA (e.g the epf transport or vhost bus).
> >>>>>> This is about connecting two different HW systems both running Linux and
> >>>>>> doesn't necessarily involve virtualization.
> >>>>> Right, this is something similar to VOP
> >>>>> (Documentation/misc-devices/mic/mic_overview.rst). The different is the
> >>>>> hardware I guess and VOP use userspace application to implement the device.
> >>>> I'd also like to point out, this series tries to have communication between
> >>>> two
> >>>> SoCs in vendor agnostic way. Since this series solves for 2 usecases (PCIe
> >>>> RC<->EP and NTB), for the NTB case it directly plugs into NTB framework and
> >>>> any
> >>>> of the HW in NTB below should be able to use a virtio-vhost communication
> >>>>
> >>>> #ls drivers/ntb/hw/
> >>>> amdÂ epfÂ idtÂ intelÂ mscc
> >>>>
> >>>> And similarly for the PCIe RC<->EP communication, this adds a generic endpoint
> >>>> function driver and hence any SoC that supports configurable PCIe endpoint can
> >>>> use virtio-vhost communication
> >>>>
> >>>> # ls drivers/pci/controller/dwc/*ep*
> >>>> drivers/pci/controller/dwc/pcie-designware-ep.c
> >>>> drivers/pci/controller/dwc/pcie-uniphier-ep.c
> >>>> drivers/pci/controller/dwc/pci-layerscape-ep.c
> >>>
> >>> Thanks for those backgrounds.
> >>>
> >>>
> >>>>>> ÂÂÂ So there is no guest or host as in
> >>>>>> virtualization but two entirely different systems connected via PCIe cable,
> >>>>>> one
> >>>>>> acting as guest and one as host. So one system will provide virtio
> >>>>>> functionality reserving memory for virtqueues and the other provides vhost
> >>>>>> functionality providing a way to access the virtqueues in virtio memory.
> >>>>>> One is
> >>>>>> source and the other is sink and there is no intermediate entity. (vhost was
> >>>>>> probably intermediate entity in virtualization?)
> >>>>> (Not a native English speaker) but "vhost" could introduce some confusion for
> >>>>> me since it was use for implementing virtio backend for userspace drivers. I
> >>>>> guess "vringh" could be better.
> >>>> Initially I had named this vringh but later decided to choose vhost instead of
> >>>> vringh. vhost is still a virtio backend (not necessarily userspace) though it
> >>>> now resides in an entirely different system. Whatever virtio is for a frontend
> >>>> system, vhost can be that for a backend system. vring can be for accessing
> >>>> virtqueue and can be used either in frontend or backend.
> >>>
> >>> Ok.
> >>>
> >>>
> >>>>>>> Have you considered to implement these through vDPA?
> >>>>>> IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg
> >>>>>> driver
> >>>>>> or vhost net driver is not provided.
> >>>>>>
> >>>>>> The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
> >>>>>> (usecase2 above),
> >>>>> I see.
> >>>>>
> >>>>>
> >>>>>> ÂÂÂ all the boards run Linux. The middle board provides NTB
> >>>>>> functionality and board on either side provides virtio/vhost
> >>>>>> functionality and
> >>>>>> transfer data using rpmsg.
> >>>>> So I wonder whether it's worthwhile for a new bus. Can we use the existed
> >>>>> virtio-bus/drivers? It might work as, except for the epf transport, we can
> >>>>> introduce a epf "vhost" transport driver.
> >>>> IMHO we'll need two buses one for frontend and other for backend because the
> >>>> two components can then co-operate/interact with each other to provide a
> >>>> functionality. Though both will seemingly provide similar callbacks, they are
> >>>> both provide symmetrical or complimentary funcitonality and need not be
> >>>> same or
> >>>> identical.
> >>>>
> >>>> Having the same bus can also create sequencing issues.
> >>>>
> >>>> If you look at virtio_dev_probe() of virtio_bus
> >>>>
> >>>> device_features = dev->config->get_features(dev);
> >>>>
> >>>> Now if we use same bus for both front-end and back-end, both will try to
> >>>> get_features when there has been no set_features. Ideally vhost device should
> >>>> be initialized first with the set of features it supports. Vhost and virtio
> >>>> should use "status" and "features" complimentarily and not identically.
> >>>
> >>> Yes, but there's no need for doing status/features passthrough in epf vhost
> >>> drivers.b
> >>>
> >>>
> >>>> virtio device (or frontend) cannot be initialized before vhost device (or
> >>>> backend) gets initialized with data such as features. Similarly vhost
> >>>> (backend)
> >>>> cannot access virqueues or buffers before virtio (frontend) sets
> >>>> VIRTIO_CONFIG_S_DRIVER_OK whereas that requirement is not there for virtio as
> >>>> the physical memory for virtqueues are created by virtio (frontend).
> >>>
> >>> epf vhost drivers need to implement two devices: vhost(vringh) device and
> >>> virtio device (which is a mediated device). The vhost(vringh) device is doing
> >>> feature negotiation with the virtio device via RC/EP or NTB. The virtio device
> >>> is doing feature negotiation with local virtio drivers. If there're feature
> >>> mismatch, epf vhost drivers and do mediation between them.
> >> Here epf vhost should be initialized with a set of features for it to negotiate
> >> either as vhost device or virtio device no? Where should the initial feature
> >> set for epf vhost come from?
> >
> >
> > I think it can work as:
> >
> > 1) Having an initial features (hard coded in the code) set X in epf vhost
> > 2) Using this X for both virtio device and vhost(vringh) device
> > 3) local virtio driver will negotiate with virtio device with feature set Y
> > 4) remote virtio driver will negotiate with vringh device with feature set Z
> > 5) mediate between feature Y and feature Z since both Y and Z are a subset of X
> >
> >
>
> okay. I'm also thinking if we could have configfs for configuring this. Anyways
> we could find different approaches of configuring this.
> >>>
> >>>>> It will have virtqueues but only used for the communication between itself
> >>>>> and
> >>>>> uppter virtio driver. And it will have vringh queues which will be probe by
> >>>>> virtio epf transport drivers. And it needs to do datacopy between
> >>>>> virtqueue and
> >>>>> vringh queues.
> >>>>>
> >>>>> It works like:
> >>>>>
> >>>>> virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh
> >>>>> queue/epf>
> >>>>>
> >>>>> The advantages is that there's no need for writing new buses and drivers.
> >>>> I think this will work however there is an addtional copy between vringh queue
> >>>> and virtqueue,
> >>>
> >>> I think not? E.g in use case 1), if we stick to virtio bus, we will have:
> >>>
> >>> virtio-rpmsg (EP) <- virtio ring(1) -> epf vhost driver (EP) <- virtio ring(2)
> >>> -> virtio pci (RC) <-> virtio rpmsg (RC)
> >> IIUC epf vhost driver (EP) will access virtio ring(2) using vringh?
> >
> >
> > Yes.
> >
> >
> >> And virtio
> >> ring(2) is created by virtio pci (RC).
> >
> >
> > Yes.
> >
> >
> >>> What epf vhost driver did is to read from virtio ring(1) about the buffer len
> >>> and addr and them DMA to Linux(RC)?
> >> okay, I made some optimization here where vhost-rpmsg using a helper writes a
> >> buffer from rpmsg's upper layer directly to remote Linux (RC) as against here
> >> were it has to be first written to virtio ring (1).
> >>
> >> Thinking how this would look for NTB
> >> virtio-rpmsg (HOST1) <- virtio ring(1) -> NTB(HOST1) <-> NTB(HOST2)Â <- virtio
> >> ring(2) -> virtio-rpmsg (HOST2)
> >>
> >> Here the NTB(HOST1) will access the virtio ring(2) using vringh?
> >
> >
> > Yes, I think so it needs to use vring to access virtio ring (1) as well.
>
> NTB(HOST1) and virtio ring(1) will be in the same system. So it doesn't have to
> use vring. virtio ring(1) is by the virtio device the NTB(HOST1) creates.
> >
> >
> >>
> >> Do you also think this will work seamlessly with virtio_net.c, virtio_blk.c?
> >
> >
> > Yes.
>
> okay, I haven't looked at this but the backend of virtio_blk should access an
> actual storage device no?
> >
> >
> >>
> >> I'd like to get clarity on two things in the approach you suggested, one is
> >> features (since epf vhost should ideally be transparent to any virtio driver)
> >
> >
> > We can have have an array of pre-defined features indexed by virtio device id
> > in the code.
> >
> >
> >> and the other is how certain inputs to virtio device such as number of buffers
> >> be determined.
> >
> >
> > We can start from hard coded the value like 256, or introduce some API for user
> > to change the value.
> >
> >
> >>
> >> Thanks again for your suggestions!
> >
> >
> > You're welcome.
> >
> > Note that I just want to check whether or not we can reuse the virtio
> > bus/driver. It's something similar to what you proposed in Software Layering
> > but we just replace "vhost core" with "virtio bus" and move the vhost core
> > below epf/ntb/platform transport.
>
> Got it. My initial design was based on my understanding of your comments [1].
>
> I'll try to create something based on your proposed design here.

Based on the above conversation it seems like I should wait for another revision
of this set before reviewing the RPMSG part. Please confirm that my
understanding is correct.

Thanks,
Mathieu

>
> Regards
> Kishon
>
> [1] ->
> https://lore.kernel.org/linux-pci/59982499-0fc1-2e39-9ff9-993fb4dd7dcc@xxxxxxxxxx/
> >
> > Thanks
> >
> >
> >>
> >> Regards
> >> Kishon
> >>
> >>>
> >>>> in some cases adds latency because of forwarding interrupts
> >>>> between vhost and virtio driver, vhost drivers providing features (which means
> >>>> it has to be aware of which virtio driver will be connected).
> >>>> virtio drivers (front end) generally access the buffers from it's local memory
> >>>> but when in backend it can access over MMIO (like PCI EPF or NTB) or
> >>>> userspace.
> >>>>> Does this make sense?
> >>>> Two copies in my opinion is an issue but lets get others opinions as well.
> >>>
> >>> Sure.
> >>>
> >>>
> >>>> Thanks for your suggestions!
> >>>
> >>> You're welcome.
> >>>
> >>> Thanks
> >>>
> >>>
> >>>> Regards
> >>>> Kishon
> >>>>
> >>>>> Thanks
> >>>>>
> >>>>>
> >>>>>> Thanks
> >>>>>> Kishon
> >>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>>
> >>>>>>>>> Kishon Vijay Abraham I (22):
> >>>>>>>>> ÂÂÂÂÂ vhost: Make _feature_ bits a property of vhost device
> >>>>>>>>> ÂÂÂÂÂ vhost: Introduce standard Linux driver model in VHOST
> >>>>>>>>> ÂÂÂÂÂ vhost: Add ops for the VHOST driver to configure VHOST device
> >>>>>>>>> ÂÂÂÂÂ vringh: Add helpers to access vring in MMIO
> >>>>>>>>> ÂÂÂÂÂ vhost: Add MMIO helpers for operations on vhost virtqueue
> >>>>>>>>> ÂÂÂÂÂ vhost: Introduce configfs entry for configuring VHOST
> >>>>>>>>> ÂÂÂÂÂ virtio_pci: Use request_threaded_irq() instead of request_irq()
> >>>>>>>>> ÂÂÂÂÂ rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
> >>>>>>>>> ÂÂÂÂÂÂÂ reading messages
> >>>>>>>>> ÂÂÂÂÂ rpmsg: Introduce configfs entry for configuring rpmsg
> >>>>>>>>> ÂÂÂÂÂ rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
> >>>>>>>>> ÂÂÂÂÂ rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
> >>>>>>>>> ÂÂÂÂÂÂÂ rpmsg_internal.h
> >>>>>>>>> ÂÂÂÂÂ virtio: Add ops to allocate and free buffer
> >>>>>>>>> ÂÂÂÂÂ rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
> >>>>>>>>> ÂÂÂÂÂÂÂ virtio_free_buffer()
> >>>>>>>>> ÂÂÂÂÂ rpmsg: Add VHOST based remote processor messaging bus
> >>>>>>>>> ÂÂÂÂÂ samples/rpmsg: Setup delayed work to send message
> >>>>>>>>> ÂÂÂÂÂ samples/rpmsg: Wait for address to be bound to rpdev for sending
> >>>>>>>>> ÂÂÂÂÂÂÂ message
> >>>>>>>>> ÂÂÂÂÂ rpmsg.txt: Add Documentation to configure rpmsg using configfs
> >>>>>>>>> ÂÂÂÂÂ virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe
> >>>>>>>>> Endpoint
> >>>>>>>>> ÂÂÂÂÂÂÂ device
> >>>>>>>>> ÂÂÂÂÂ PCI: endpoint: Add EP function driver to provide VHOST interface
> >>>>>>>>> ÂÂÂÂÂ NTB: Add a new NTB client driver to implement VIRTIO functionality
> >>>>>>>>> ÂÂÂÂÂ NTB: Add a new NTB client driver to implement VHOST functionality
> >>>>>>>>> ÂÂÂÂÂ NTB: Describe the ntb_virtio and ntb_vhost client in the
> >>>>>>>>> documentation
> >>>>>>>>>
> >>>>>>>>> ÂÂÂÂ Documentation/driver-api/ntb.rstÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 11 +
> >>>>>>>>> ÂÂÂÂ Documentation/rpmsg.txtÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 56 +
> >>>>>>>>> ÂÂÂÂ drivers/ntb/KconfigÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 18 +
> >>>>>>>>> ÂÂÂÂ drivers/ntb/MakefileÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 2 +
> >>>>>>>>> ÂÂÂÂ drivers/ntb/ntb_vhost.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 776 +++++++++++
> >>>>>>>>> ÂÂÂÂ drivers/ntb/ntb_virtio.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 853 ++++++++++++
> >>>>>>>>> ÂÂÂÂ drivers/ntb/ntb_virtio.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 56 +
> >>>>>>>>> ÂÂÂÂ drivers/pci/endpoint/functions/KconfigÂÂÂÂÂÂÂ |ÂÂ 11 +
> >>>>>>>>> ÂÂÂÂ drivers/pci/endpoint/functions/MakefileÂÂÂÂÂÂ |ÂÂÂ 1 +
> >>>>>>>>> ÂÂÂÂ .../pci/endpoint/functions/pci-epf-vhost.cÂÂÂ | 1144
> >>>>>>>>> ++++++++++++++++
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/KconfigÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 10 +
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/MakefileÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 3 +-
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/rpmsg_cfs.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 394 ++++++
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/rpmsg_core.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 7 +
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/rpmsg_internal.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 136 ++
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/vhost_rpmsg_bus.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | 1151
> >>>>>>>>> +++++++++++++++++
> >>>>>>>>> ÂÂÂÂ drivers/rpmsg/virtio_rpmsg_bus.cÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 184 ++-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/KconfigÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 1 +
> >>>>>>>>> ÂÂÂÂ drivers/vhost/MakefileÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 2 +-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/net.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 10 +-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/scsi.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 24 +-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/test.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 17 +-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/vdpa.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 2 +-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/vhost.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 730 ++++++++++-
> >>>>>>>>> ÂÂÂÂ drivers/vhost/vhost_cfs.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 341 +++++
> >>>>>>>>> ÂÂÂÂ drivers/vhost/vringh.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 332 +++++
> >>>>>>>>> ÂÂÂÂ drivers/vhost/vsock.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 20 +-
> >>>>>>>>> ÂÂÂÂ drivers/virtio/KconfigÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 9 +
> >>>>>>>>> ÂÂÂÂ drivers/virtio/MakefileÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 1 +
> >>>>>>>>> ÂÂÂÂ drivers/virtio/virtio_pci_common.cÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 25 +-
> >>>>>>>>> ÂÂÂÂ drivers/virtio/virtio_pci_epf.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â 670 ++++++++++
> >>>>>>>>> ÂÂÂÂ include/linux/mod_devicetable.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 6 +
> >>>>>>>>> ÂÂÂÂ include/linux/rpmsg.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 6 +
> >>>>>>>>> ÂÂÂÂ {drivers/vhost => include/linux}/vhost.hÂÂÂÂÂ |Â 132 +-
> >>>>>>>>> ÂÂÂÂ include/linux/virtio.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 3 +
> >>>>>>>>> ÂÂÂÂ include/linux/virtio_config.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 42 +
> >>>>>>>>> ÂÂÂÂ include/linux/vringh.hÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ 46 +
> >>>>>>>>> ÂÂÂÂ samples/rpmsg/rpmsg_client_sample.cÂÂÂÂÂÂÂÂÂÂ |ÂÂ 32 +-
> >>>>>>>>> ÂÂÂÂ tools/virtio/virtio_test.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ 2 +-
> >>>>>>>>> ÂÂÂÂ 39 files changed, 7083 insertions(+), 183 deletions(-)
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/ntb/ntb_vhost.c
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/ntb/ntb_virtio.c
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/ntb/ntb_virtio.h
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/rpmsg/rpmsg_cfs.c
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/vhost/vhost_cfs.c
> >>>>>>>>> ÂÂÂÂ create mode 100644 drivers/virtio/virtio_pci_epf.c
> >>>>>>>>> ÂÂÂÂ rename {drivers/vhost => include/linux}/vhost.h (66%)
> >>>>>>>>>
> >>>>>>>>> --Â
> >>>>>>>>> 2.17.1
> >>>>>>>>>
> >

Next message: Pawan Gupta: "Re: [PATCH] x86/bugs/multihit: Fix mitigation reporting when KVM is not in use"
Previous message: Matthew Wilcox: "Re: strace of io_uring events?"
In reply to: Jason Wang: "Re: [RFC PATCH 00/22] Enhance VHOST to enable SoC-to-SoC communication"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]