Re: PCIe RC\EP virtio rdma solution discussion.

[Shunsuke Mie]

Thanks for organizing the discussion.

On 2023/02/08 4:45, Frank Li wrote:
> From: Frank Li<Frank.li@xxxxxxx>
>
> Recently more and more people are interested in PCI RC and EP connection,
> especially network usage cases. I upstreamed a vntb solution last year.
> But the transfer speed is not good enough. I initialized a discussion
> athttps://lore.kernel.org/imx/d098a631-9930-26d3-48f3-8f95386c8e50@xxxxxx/T/#t
I've investigated the vntb + ntbnet device that uses ntb transfer. Is it
difficult to adapt the eDMA to the ntb transfer? It is likely one of the
solutions for the performance problem.
>   
>    ┌─────────────────────────────────┐   ┌──────────────┐
>    │                                 │   │              │
>    │                                 │   │              │
>    │   VirtQueue             RX      │   │  VirtQueue   │
>    │     TX                 ┌──┐     │   │    TX        │
>    │  ┌─────────┐           │  │     │   │ ┌─────────┐  │
>    │  │ SRC LEN ├─────┐  ┌──┤  │◄────┼───┼─┤ SRC LEN │  │
>    │  ├─────────┤     │  │  │  │     │   │ ├─────────┤  │
>    │  │         │     │  │  │  │     │   │ │         │  │
>    │  ├─────────┤     │  │  │  │     │   │ ├─────────┤  │
>    │  │         │     │  │  │  │     │   │ │         │  │
>    │  └─────────┘     │  │  └──┘     │   │ └─────────┘  │
>    │                  │  │           │   │              │
>    │     RX       ┌───┼──┘   TX      │   │    RX        │
>    │  ┌─────────┐ │   │     ┌──┐     │   │ ┌─────────┐  │
>    │  │         │◄┘   └────►│  ├─────┼───┼─┤         │  │
>    │  ├─────────┤           │  │     │   │ ├─────────┤  │
>    │  │         │           │  │     │   │ │         │  │
>    │  ├─────────┤           │  │     │   │ ├─────────┤  │
>    │  │         │           │  │     │   │ │         │  │
>    │  └─────────┘           │  │     │   │ └─────────┘  │
>    │   virtio_net           └──┘     │   │ virtio_net   │
>    │  Virtual PCI BUS   EDMA Queue   │   │              │
>    ├─────────────────────────────────┤   │              │
>    │  PCI EP Controller with eDMA    │   │  PCI Host    │
>    └─────────────────────────────────┘   └──────────────┘
>
> Basic idea is
> 	1.	Both EP and host probe virtio_net driver
> 	2.	There are two queues,  one is the EP side(EQ),  the other is the Host side.
> 	3.	EP side epf driver map Host side’s queue into EP’s space. Called HQ.
> 	4.	One working thread
> 	5.	pick one TX from EQ and RX from HQ, combine and generate EDMA requests,
> and put into the DMA TX queue.
> 	6.	Pick one RX from EQ and TX from HQ, combine and generate EDMA requests,
> and put into the DMA RX queue.
> 	7.	EDMA done irq will mark related item in EP and HQ finished.
>
> The whole transfer is zero copied and uses a DMA queue.
>
> The Shunsuke Mie implemented the above idea.
>   https://lore.kernel.org/linux-pci/CANXvt5q_qgLuAfF7dxxrqUirT_Ld4B=POCq8JcB9uPRvCGDiKg@xxxxxxxxxxxxxx/T/#t
>
>
> Similar solution posted at 2019, except use memcpy from/to PCI EP map windows.
> Using DMA should be simpler because EDMA can access the whole HOST\EP side memory space.
> https://lore.kernel.org/linux-pci/9f8e596f-b601-7f97-a98a-111763f966d1@xxxxxx/T/
>
> Solution 1 (Based on shunsuke):
>
> Both EP and Host side use virtio.
> Using EDMA to simplify data transfer and improve transfer speed.
> RDMA implement based on RoCE
> - proposal:https://lore.kernel.org/all/20220511095900.343-1-xieyongji@xxxxxxxxxxxxx/T/
> - presentation on kvm forum:https://youtu.be/Qrhv6hC_YK4
>
> Solution 2(2020, Kishon)
>
> Previoushttps://lore.kernel.org/linux-pci/20200702082143.25259-1-kishon@xxxxxx/
> EP side use vhost, RC side use virtio.
> I don’t think anyone works on this thread now.
> If using eDMA, it needs both sides to have a transfer queue.
> I don't know how to easily implement it on the vhost side.
We had implemented this solution at the design stage of our proposal.
This solution has to prepare a network device and register to the kernel
from scratch for the endpoint. There is a lot of duplicated code, so we
think the solution 1 is better, as Frank said.
> Solution 3(I am working on)
>
> Implement infiniband rdma driver at both EP and RC side.
> EP side build EDMA hardware queue based on EP/RC side’s send and receive
> queue and when eDMA finished, write status to complete queue for both EP/RC
> side. Use ipoib implement network transfer.
The new InfiniBand device has to implement an InfiniBand network layer.
I think it is overengineered for this peer-to-peer communication. In
addition, the driver of the InfiniBand device should be implemented or
emulate the existing InfiniBand device to use the upstream driver. We
want to reduce the cost of implementation and maintenance.
> The whole upstream effort is quite huge for these. I don’t want to waste
> time and efforts because direction is wrong.
>
> I think Solution 1 is an easy path.
Best,

Shunsuke.