Re: [RFC PATCH 00/22] Enhance VHOST to enable SoC-to-SoC communication

From: Jason Wang
Date: Fri Jul 03 2020 - 03:16:40 EST


On 2020/7/2 äå9:35, Kishon Vijay Abraham I wrote:
Hi Jason,

On 7/2/2020 3:40 PM, Jason Wang wrote:
On 2020/7/2 äå5:51, Michael S. Tsirkin wrote:
On Thu, Jul 02, 2020 at 01:51:21PM +0530, Kishon Vijay Abraham I wrote:
This series enhances Linux Vhost support to enable SoC-to-SoC
communication over MMIO. This series enables rpmsg communication between
two SoCs using both PCIe RC<->EP and HOST1-NTB-HOST2

1) Modify vhost to use standard Linux driver model
2) Add support in vring to access virtqueue over MMIO
3) Add vhost client driver for rpmsg
4) Add PCIe RC driver (uses virtio) and PCIe EP driver (uses vhost) for
ÂÂÂ rpmsg communication between two SoCs connected to each other
5) Add NTB Virtio driver and NTB Vhost driver for rpmsg communication
ÂÂÂ between two SoCs connected via NTB
6) Add configfs to configure the components

UseCase1 :

 VHOST RPMSG VIRTIO RPMSG
ÂÂÂÂÂÂ +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +
ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
+-----v------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------v-------+
|ÂÂ LinuxÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ LinuxÂÂÂ |
| Endpoint | | Root Complex |
|ÂÂÂÂÂÂÂÂÂÂÂ <----------------->ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
|ÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
|ÂÂÂ SOC1ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ SOC2ÂÂÂÂ |
+------------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +--------------+

UseCase 2:

ÂÂÂÂÂ VHOST RPMSGÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VIRTIO RPMSG
ÂÂÂÂÂÂÂÂÂÂ +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂ +------v------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------v------+
ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂ |ÂÂÂ HOST1ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂ HOST2ÂÂÂ |
ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂ +------^------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------^------+
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
+---------------------------------------------------------------------+
|Â +------v------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------v------+Â |
|Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
|Â |ÂÂÂÂ EPÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂ EPÂÂÂÂÂ |Â |
|Â | CONTROLLER1 |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | CONTROLLER2 |Â |
|Â |ÂÂÂÂÂÂÂÂÂÂÂÂ <----------------------------------->ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
|Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
|Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
|Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â SoC With Multiple EP InstancesÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
|Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â (Configured using NTB Function)Â |ÂÂÂÂÂÂÂÂÂÂÂÂ |Â |
|Â +-------------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +-------------+Â |
+---------------------------------------------------------------------+

Software Layering:

The high-level SW layering should look something like below. This series
adds support only for RPMSG VHOST, however something similar should be
done for net and scsi. With that any vhost device (PCI, NTB, Platform
device, user) can use any of the vhost client driver.


ÂÂÂÂ +----------------+Â +-----------+Â +------------+Â +----------+
ÂÂÂÂ |Â RPMSG VHOSTÂÂ |Â | NET VHOST |Â | SCSI VHOST |Â |ÂÂÂ XÂÂÂÂ |
ÂÂÂÂ +-------^--------+Â +-----^-----+Â +-----^------+Â +----^-----+
ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂ |
+-----------v-----------------v--------------v--------------v----------+
|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ VHOST COREÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
+--------^---------------^--------------------^------------------^-----+
ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
ÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
+--------v-------+Â +----v------+Â +----------v----------+Â +----v-----+
|Â PCI EPF VHOST |Â | NTB VHOST |Â |PLATFORM DEVICE VHOST|Â |ÂÂÂ XÂÂÂÂ |
+----------------+Â +-----------+Â +---------------------+Â +----------+

This was initially proposed here [1]

[1] -> https://lore.kernel.org/r/2cf00ec4-1ed6-f66e-6897-006d1a5b6390@xxxxxx
I find this very interesting. A huge patchset so will take a bit
to review, but I certainly plan to do that. Thanks!

Yes, it would be better if there's a git branch for us to have a look.
I've pushed the branch
https://github.com/kishon/linux-wip.git vhost_rpmsg_pci_ntb_rfc


Thanks


Btw, I'm not sure I get the big picture, but I vaguely feel some of the work is
duplicated with vDPA (e.g the epf transport or vhost bus).
This is about connecting two different HW systems both running Linux and
doesn't necessarily involve virtualization.


Right, this is something similar to VOP (Documentation/misc-devices/mic/mic_overview.rst). The different is the hardware I guess and VOP use userspace application to implement the device.


So there is no guest or host as in
virtualization but two entirely different systems connected via PCIe cable, one
acting as guest and one as host. So one system will provide virtio
functionality reserving memory for virtqueues and the other provides vhost
functionality providing a way to access the virtqueues in virtio memory. One is
source and the other is sink and there is no intermediate entity. (vhost was
probably intermediate entity in virtualization?)


(Not a native English speaker) but "vhost" could introduce some confusion for me since it was use for implementing virtio backend for userspace drivers. I guess "vringh" could be better.



Have you considered to implement these through vDPA?
IIUC vDPA only provides an interface to userspace and an in-kernel rpmsg driver
or vhost net driver is not provided.

The HW connection looks something like https://pasteboard.co/JfMVVHC.jpg
(usecase2 above),


I see.


all the boards run Linux. The middle board provides NTB
functionality and board on either side provides virtio/vhost functionality and
transfer data using rpmsg.


So I wonder whether it's worthwhile for a new bus. Can we use the existed virtio-bus/drivers? It might work as, except for the epf transport, we can introduce a epf "vhost" transport driver.

It will have virtqueues but only used for the communication between itself and uppter virtio driver. And it will have vringh queues which will be probe by virtio epf transport drivers. And it needs to do datacopy between virtqueue and vringh queues.

It works like:

virtio drivers <- virtqueue/virtio-bus -> epf vhost drivers <- vringh queue/epf>

The advantages is that there's no need for writing new buses and drivers.

Does this make sense?

Thanks



Thanks
Kishon

Thanks


Kishon Vijay Abraham I (22):
ÂÂ vhost: Make _feature_ bits a property of vhost device
ÂÂ vhost: Introduce standard Linux driver model in VHOST
ÂÂ vhost: Add ops for the VHOST driver to configure VHOST device
ÂÂ vringh: Add helpers to access vring in MMIO
ÂÂ vhost: Add MMIO helpers for operations on vhost virtqueue
ÂÂ vhost: Introduce configfs entry for configuring VHOST
ÂÂ virtio_pci: Use request_threaded_irq() instead of request_irq()
ÂÂ rpmsg: virtio_rpmsg_bus: Disable receive virtqueue callback when
ÂÂÂÂ reading messages
ÂÂ rpmsg: Introduce configfs entry for configuring rpmsg
ÂÂ rpmsg: virtio_rpmsg_bus: Add Address Service Notification support
ÂÂ rpmsg: virtio_rpmsg_bus: Move generic rpmsg structure to
ÂÂÂÂ rpmsg_internal.h
ÂÂ virtio: Add ops to allocate and free buffer
ÂÂ rpmsg: virtio_rpmsg_bus: Use virtio_alloc_buffer() and
ÂÂÂÂ virtio_free_buffer()
ÂÂ rpmsg: Add VHOST based remote processor messaging bus
ÂÂ samples/rpmsg: Setup delayed work to send message
ÂÂ samples/rpmsg: Wait for address to be bound to rpdev for sending
ÂÂÂÂ message
ÂÂ rpmsg.txt: Add Documentation to configure rpmsg using configfs
ÂÂ virtio_pci: Add VIRTIO driver for VHOST on Configurable PCIe Endpoint
ÂÂÂÂ device
ÂÂ PCI: endpoint: Add EP function driver to provide VHOST interface
ÂÂ NTB: Add a new NTB client driver to implement VIRTIO functionality
ÂÂ NTB: Add a new NTB client driver to implement VHOST functionality
ÂÂ NTB: Describe the ntb_virtio and ntb_vhost client in the documentation

 Documentation/driver-api/ntb.rst | 11 +
 Documentation/rpmsg.txt | 56 +
 drivers/ntb/Kconfig | 18 +
 drivers/ntb/Makefile | 2 +
 drivers/ntb/ntb_vhost.c | 776 +++++++++++
 drivers/ntb/ntb_virtio.c | 853 ++++++++++++
 drivers/ntb/ntb_virtio.h | 56 +
 drivers/pci/endpoint/functions/Kconfig | 11 +
 drivers/pci/endpoint/functions/Makefile | 1 +
 .../pci/endpoint/functions/pci-epf-vhost.c | 1144 ++++++++++++++++
 drivers/rpmsg/Kconfig | 10 +
 drivers/rpmsg/Makefile | 3 +-
 drivers/rpmsg/rpmsg_cfs.c | 394 ++++++
 drivers/rpmsg/rpmsg_core.c | 7 +
 drivers/rpmsg/rpmsg_internal.h | 136 ++
 drivers/rpmsg/vhost_rpmsg_bus.c | 1151 +++++++++++++++++
 drivers/rpmsg/virtio_rpmsg_bus.c | 184 ++-
 drivers/vhost/Kconfig | 1 +
 drivers/vhost/Makefile | 2 +-
 drivers/vhost/net.c | 10 +-
 drivers/vhost/scsi.c | 24 +-
 drivers/vhost/test.c | 17 +-
 drivers/vhost/vdpa.c | 2 +-
 drivers/vhost/vhost.c | 730 ++++++++++-
 drivers/vhost/vhost_cfs.c | 341 +++++
 drivers/vhost/vringh.c | 332 +++++
 drivers/vhost/vsock.c | 20 +-
 drivers/virtio/Kconfig | 9 +
 drivers/virtio/Makefile | 1 +
 drivers/virtio/virtio_pci_common.c | 25 +-
 drivers/virtio/virtio_pci_epf.c | 670 ++++++++++
 include/linux/mod_devicetable.h | 6 +
 include/linux/rpmsg.h | 6 +
 {drivers/vhost => include/linux}/vhost.h | 132 +-
 include/linux/virtio.h | 3 +
 include/linux/virtio_config.h | 42 +
 include/linux/vringh.h | 46 +
 samples/rpmsg/rpmsg_client_sample.c | 32 +-
 tools/virtio/virtio_test.c | 2 +-
 39 files changed, 7083 insertions(+), 183 deletions(-)
 create mode 100644 drivers/ntb/ntb_vhost.c
 create mode 100644 drivers/ntb/ntb_virtio.c
 create mode 100644 drivers/ntb/ntb_virtio.h
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vhost.c
 create mode 100644 drivers/rpmsg/rpmsg_cfs.c
 create mode 100644 drivers/rpmsg/vhost_rpmsg_bus.c
 create mode 100644 drivers/vhost/vhost_cfs.c
 create mode 100644 drivers/virtio/virtio_pci_epf.c
 rename {drivers/vhost => include/linux}/vhost.h (66%)

--
2.17.1