Re: [PATCH v2 0/3] PCI: endpoint: Support hardware-owned MSI-X table and PBA

[Koichiro Den]

On Mon, Aug 31, 2026 at 12:24:55PM +0200, Niklas Cassel wrote:
> Hello Koichiro,
> 
> On Mon, Aug 31, 2026 at 12:19:45AM +0900, Koichiro Den wrote:
> > Hi,
> > 
> > Some DWC endpoint controllers keep the MSI-X Table and PBA at fixed
> > locations in a reserved BAR. On RK3588, the MSI-X doorbell uses this
> > hardware-owned layout, as discussed in e.g. [1].
> > 
> > The reserved-region types for the Table and PBA were added with the
> > Tegra194 description. The MSI-X setup API, however, still takes only the
> > Table BAR and offset and assumes that the PBA immediately follows it in
> > the same BAR.
> > 
> > This series passes the complete layout to pci_epc_set_msix() and adds
> > pci_epc_get_hw_msix_layout() to expose a hardware-owned layout. The core
> > does not select it automatically. The choice stays with the EPF.
> > 
> > A hardware-owned layout is not a direct replacement for every caller.
> > pci-epf-ntb, for example, reads host-programmed Table entries from its
> > own BAR to set up peer outbound mappings.
> > 
> > The series also fixes an interrupt-type mismatch in vNTB. ntb_hw_epf can
> > select MSI-X, while pci-epf-vntb currently configures and raises only
> > MSI. On RK3588, selecting the fixed BAR4 layout also selects the DWC
> > MSI-X doorbell. EPF-owned layouts continue to use the regular path.
> > 
> > In short:
> > 
> > - pci-epf-vntb gains MSI-X support and uses the hardware-owned layout
> >   when available.
> > - pci-epf-test, pci-epf-ntb, and the NVMe PCI EPF keep their EPF-owned
> >   layouts. This avoids unnecessary changes and reduces regression risk.
> >   They can use a hardware-owned layout later if/when needed.
> > 
> > [1] https://lore.kernel.org/r/aY2q80zeRKSRO21H@fedora
> 

Hello Niklas,

> 
> Perhaps you could improve the cover letter to more clearly state why you
> are doing this change.

Thanks, that's a fair point, the cover letter does not explain the
motivation clearly enough.

> 
> Some guesses:
> - Better performance. We avoid the need to map + unmap the MSI target
>   address using an iATU each time we raise an MSI-X. We also avoid the
>   need to flush posted write before unmap.
>   Is there any performance difference? If so, it would be nice with some
>   numbers.
> - Allows more concurrent I/Os. By not using an iATU when raising an MSI-X,
>   we have one more iATU available, so we can have one more outstanding I/O.
> - Less waste of BAR space. (Since the MSI-X table and PBA already
>   always takes up space in one of the BARs, it is wasteful to have
>   the EPF drive duplicate it in another BAR.)

This is not driven by a performance target, so I do not have numbers. The
immediate issue is correctness. ntb_hw_epf can select MSI-X, but pci-epf-vntb
currently configures and raises MSI. I want that MSI-X path to work on RK3588
with the controller doorbell. I will make it clear in the next cover letter
that, with this series, a doorbell sent from ntb_tool on the pci-epf-vntb side
of RK3588 reaches ntb_tool on the ntb_hw_epf host side when MSI-X is selected.
I believe that is the simplest E2E test showing what this series fixes.

For pci-epf-vntb this is a new MSI-X path, not a change to an existing working
path, so it seemed like a safe first user of the hardware-owned layout.
Honestly, adding a new MSI-X path to pci-epf-vntb while allocating another
Table/PBA in an EPF-owned BAR on RK3588 seemd rather odd to me.

So, avoiding the temporary iATU mapping and duplicate Table/PBA is good, but for
me performance was not the goal for my RK3588 case.

> 
> 
> 
> Personally, I don't see why we should only change pci-epf-vntb to use
> the hardware-owned layout when available. I don't see why we would not

I initially considered making the hardware-owned layout automatic for every EPF.
pci-epf-ntb (not vNTB!) is the reason I did not. It currently reads
host-programmed MSI-X entries through epf_bar[] backing and uses them to set up
peer outbound mappings. A hardware-owned Table has no such backing, and the
current EPC API has no generic way to read those entries. I suspect that
supporting it there would require a broader design change. Also, I do not have
hardware for the pci-epf-ntb bridge configuration.

pci-epf-test and nvmet-pci-epf should be easier to convert. However, converting
only those would not remove the per-layout handling while pci-epf-ntb still uses
an EPF-owned Table. I would prefer to handle those separately when they can be
properly tested. This is what I meant by this part of the cover letter:

     [...]

     layouts. This avoids unnecessary changes and reduces regression risk.
     They can use a hardware-owned layout later if/when needed.

> want to change pci-epf-test, pci-epf-ntb, and nvmet-pci-epf as well.
> (If the EPC defines a HW defined MSI-X table + PBA, why not always
> use that? If there is no HW defined MSI-X table + PBA, let the EPF
> put the MSI-X table in any BAR it likes.)
> 
> 
> I understand that you introduce dw_pcie_ep_msix_layout_is_hw_owned()
> because you want an EPF driver optionally use the HW defined table.
> But if all EPF drivers always use the HW defined table if available, I think
> you can avoid introducing this helper, and let rockchip_pcie_raise_irq()
> unconditionally call dw_pcie_ep_raise_msix_irq_doorbell() for case PCI_IRQ_MSIX.

Yes, under that assumption I agree. With the current mix, however, the Rockchip
callback still has to choose the helper according to the selected layout. That
is why the DWC code records which layout was selected.

I will update the cover letter to state the motivation and this boundary more
directly. Does keeping the layout choice with each EPF for this reason sound
reasonable?

Best regards,
Koichiro

> 
> See e.g. drivers/pci/controller/dwc/pci-layerscape-ep.c which already calls
> dw_pcie_ep_raise_msix_irq_doorbell() unconditionally for case PCI_IRQ_MSIX.
> 
> 
> Kind regards,
> Niklas