RE: [RFC] Proposal of QEMU PCI Endpoint test environment

[Thanos Makatos]

> -----Original Message-----
> From: Shunsuke Mie <mie@xxxxxxxxxx>
> Sent: Friday, October 6, 2023 12:51 PM
> To: Mattias Nissler <mnissler@xxxxxxxxxxxx>
> Cc: cz172638@xxxxxxxxx; bhelgaas@xxxxxxxxxx; Jagannathan Raman
> <jag.raman@xxxxxxxxxx>; kishon@xxxxxxxxxx; kvijayab@xxxxxxx;
> kw@xxxxxxxxx; levon@xxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; linux-
> pci@xxxxxxxxxxxxxxx; lpieralisi@xxxxxxxxxx; manivannan.sadhasivam@xxxxxxxxxx;
> Marcel Apfelbaum <marcel.apfelbaum@xxxxxxxxx>; Michael S. Tsirkin
> <mst@xxxxxxxxxx>; Paolo Bonzini <pbonzini@xxxxxxxxxx>; qemu-
> devel@xxxxxxxxxx; robh@xxxxxxxxxx; Thanos Makatos
> <thanos.makatos@xxxxxxxxxxx>; vaishnav.a@xxxxxx; William Henderson
> <william.henderson@xxxxxxxxxxx>
> Subject: Re: [RFC] Proposal of QEMU PCI Endpoint test environment
> 
> 
> On 2023/10/05 16:02, Mattias Nissler wrote:
> > On Thu, Oct 5, 2023 at 3:31 AM Shunsuke Mie <mie@xxxxxxxxxx> wrote:
> >> Hi Jiri, Mattias and all.
> >>
> >> 2023年10月4日(水) 16:36 Mattias Nissler <mnissler@xxxxxxxxxxxx>:
> >>>> hi shunsuke, all,
> >>>> what about vfio-user + qemu?
> >> Thank you for the suggestion.
> >>
> >>> FWIW, I have had some good success using VFIO-user to bridge software
> components to hardware designs. For the most part, I have been hooking up
> software endpoint models to hardware design components speaking the PCIe
> transaction layer protocol. The central piece you need is a way to translate
> between the VFIO-user protocol and PCIe transaction layer messages, basically
> converting ECAM accesses, memory accesses (DMA+MMIO), and interrupts
> between the two worlds. I have some code which implements the basics of that.
> It's certainly far from complete (TLP is a massive protocol), but it works well
> enough for me. I believe we should be able to open-source this if there's interest,
> let me know.
> >> It is what I want to do, but I'm not familiar with the vfio and vfio-user, and I
> have a question. QEMU has a PCI TLP communication implementation for Multi-
> process QEMU[1]. It is similar to your success.
> > I'm no qemu expert, but my understanding is that the plan is for the
> > existing multi-process QEMU implementation to eventually be
> > superseded/replaced by the VFIO-user based one (qemu folks, please
> > correct me if I'm wrong). From a functional perspective they are more
> > or less equivalent AFAICT.
> >
> The project is promising.
> 
> I found a session about the vfio adapts to Multi-process QEMU[1] in KVM
> Forun 2021, butI couldn't found some posted patches.
> If anyone knows status of this project, could you please let me know?

AFAIK the mp-qemu folk are working on continuing JJ's work to enable vfio-user client on QEMU, not sure about the timeline, Jag can you comment?

You can still play around with their forked version of QEMU (which they use to post patches), the libvfio-user documentation explains how to use it.

> 
> [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.youtube.com_watch-3Fv-
> 3DNBT8rImx3VE&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvt
> w6ogtti46atk736SI4vgsJiUKIyDE&m=lGbUrPHS1zjVVRi0P-
> ukZ0TAabgefclz26Q56PgHzKw6oxFrtqgpv1g_aDF9hXZk&s=EU-
> KA65Gk2jC3zhimGSX96Mfz3kxZnU7gqJ00F4G4DM&e=
> >> The multi-process qemu also communicates TLP over UDS. Could you let me
> know your opinion about it?
> > Note that neither multi-process qemu nor VFIO-user actually pass
> > around TLPs, but rather have their own command language to encode
> > ECAM, MMIO, DMA, interrupts etc. However, translation from/to TLP is
> > possible and works well enough in my experience.
> I agree.
> >>> One thing to note is that there are currently some limits to bridging VFIO-user
> / TLP that I haven't figured out and/or will need further work: Advanced PCIe
> concepts like PASID, ATS/PRI, SR-IOV etc. may lack equivalents on the VFIO-user
> side that would have to be filled in. The folk behind libvfio-user[2] have been very
> approachable and open to improvements in my experience though.
> >>>
> >>> If I understand correctly, the specific goal here is testing PCIe endpoint
> designs against a Linux host. What you'd need for that is a PCI host controller for
> the Linux side to talk to and then hooking up endpoints on the transaction layer.
> QEMU can simulate host controllers that work with existing Linux drivers just fine.
> Then you can put a vfio-user-pci stub device (I don't think this has landed in qemu
> yet, but you can find the code at [1]) on the simulated PCI bus which will expose
> any software interactions with the endpoint as VFIO-user protocol messages over
> unix domain socket. The piece you need to bring is a VFIO-user server that
> handles these messages. Its task is basically translating between VFIO-user and
> TLP and then injecting TLP into your hardware design.
> >> Yes, If the pci host controller you said can be implemented, I can achieve my
> goal.
> > I meant to say that the existing PCIe host controller implementations
> > in qemu can be used as is.
> Sorry, I misunderstood.
> >> To begin with, I'll investigate the vfio and libvfio-user.  Thanks!.
> >>
> >> [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.qemu.org_docs_master_system_multi-
> 2Dprocess.html&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw
> 6ogtti46atk736SI4vgsJiUKIyDE&m=lGbUrPHS1zjVVRi0P-
> ukZ0TAabgefclz26Q56PgHzKw6oxFrtqgpv1g_aDF9hXZk&s=QW5gt2SlCvGU8T20L2
> PoPEX_weidbGfnxiYPmVAEnVQ&e=
> >>
> >> Best,
> >> Shunsuke
> >>>
> >>> [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_oracle_qemu_tree_vfio-2Duser-
> 2Dp3.1&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46
> atk736SI4vgsJiUKIyDE&m=lGbUrPHS1zjVVRi0P-
> ukZ0TAabgefclz26Q56PgHzKw6oxFrtqgpv1g_aDF9hXZk&s=l8dGYrL2oJmcUoD22A
> mQqbDukmY5UA_IfSAcmdMvnPI&e=  - I believe that's the latest version,
> Jagannathan Raman will know best
> >>> [2] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_nutanix_libvfio-
> 2Duser&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46a
> tk736SI4vgsJiUKIyDE&m=lGbUrPHS1zjVVRi0P-
> ukZ0TAabgefclz26Q56PgHzKw6oxFrtqgpv1g_aDF9hXZk&s=MqK4yRxBjOVOLppnN
> k_TYpg7p5gUg2g1CW5Wt74up1E&e=
> >>>