Re: [PATCH 00/11] PCI/NTB: endpoint: packed vNTB memory windows

[Koichiro Den]

On Mon, Aug 03, 2026 at 04:00:45PM -0700, Randy Dunlap wrote:
> 
> 
> On 8/3/26 11:04 AM, Koichiro Den wrote:
> > Hi,
> > 
> > vNTB currently spends one BAR per MW. BAR count therefore limits the
> > number of MWs, even when one BAR has room for several. A large BAR,
> > especially a Resizable BAR, can instead be split into logical MWs and
> > leave more BARs for config and doorbells.
> > 
> > This series adds packed_mws, which exposes up to 16 equal-sized logical
> > MWs in one BAR. ntb_transport uses the new group operations to configure
> > them atomically.
> > 
> > The immediate use case is increasing the number of ntb_transport queues.
> > The resulting performance gain is shown in the Testing section.
> > 
> > Based on the latest pci/endpoint.
> > 
> > 
> > Why not BAR subrange mappings?
> > ==============================
> > 
> > An earlier proposal used mwN_offset and BAR subrange mappings to place
> > independently backed MWs in one BAR:
> > 
> >   https://lore.kernel.org/r/20260312165005.1148676-11-den@xxxxxxxxxxxxx/
> > 
> > BAR subrange mapping is currently implemented only by DWC, and that
> > design still needs one hardware mapping per MW in each direction. This
> > series instead requires one contiguous backing range and maps the whole
> > group once in each direction. The initial implementation intentionally
> > trades arbitrary MW placement for a constant mapping count as the MW
> > count grows. More flexible layouts can be added when a concrete use
> > case requires them.
> > 
> 
> Hi,
> To be clear, this is totally outside of the scope of any PCI or NTB
> standards or specs?  or did I overlook that part of the discussion?
> Thanks.

Hi Randy,

No, you didn't overlook anything. This isn't part of any PCI or NTB spec.
This series just proposes a new 'packed_mws' configfs knob for the
vNTB/ntb_hw_epf pair. PCI still sees a normal BAR.

Thanks,
Koichiro

> 
> > 
> > Testing
> > =======
> > 
> > The packed data path was tested with ntb_transport and ntb_netdev. An
> > `iperf3 -ub0 -l 65482 -P 2` UDP saturation run produced the following
> > results when only one Resizable BAR was available for MWs:
> > 
> >   Configuration                                   Receiver throughput
> >   ----------------------------------------------  -------------------
> >   num_mws=1, packed_mws unset (default 0)         607 Mbits/sec
> >   num_mws=2, packed_mws=2, combined channels=2    1.19 Gbits/sec
> > 
> > Meanwhile, I realized that the in-tree ntb_test.sh does not fully pass
> > with the pci-epf-vntb/ntb_hw_epf setup on the base tree. I had to apply
> > a separate collection of local PCI endpoint and NTB fixes before this
> > setup could be used for regression testing. These fixes address
> > pre-existing issues and are not prerequisites of the packed MW
> > implementation.
> > 
> > With packed_mws unset and ntb_msi_tests disabled, ntb_test.sh reported
> > the same pass/unsupported results before and after this series. One run
> > follows:
> > 
> >   $ sudo ./ntb_test.sh -r ${EP} -m 1048576 -p 28 0000:01:00.0 0001:10:00.0
> >   Starting ntb_tool tests...
> >   Running port tests on: 0000:01:00.0 / 0001:10:00.0
> >   Local port 0 with index 0 on remote host
> >   Peer port 0 with index 0 on local host
> >     Passed
> >   Running link tests on: 0000:01:00.0 / 0001:10:00.0
> >     Passed
> >   Running link tests on: 0001:10:00.0 / 0000:01:00.0
> >     Passed
> >   Running db tests on: 0000:01:00.0 / 0001:10:00.0
> >     Passed
> >   Running db tests on: 0001:10:00.0 / 0000:01:00.0
> >     Passed
> >   Running spad tests on: 0000:01:00.0 / 0001:10:00.0
> >     Passed
> >   Running spad tests on: 0001:10:00.0 / 0000:01:00.0
> >     Passed
> >   Running msg tests on: 0000:01:00.0 / 0001:10:00.0
> >     Unsupported
> >   Running msg tests on: 0001:10:00.0 / 0000:01:00.0
> >     Unsupported
> >   Running mw0 tests on: 0000:01:00.0 /  0001:10:00.0
> >     Passed
> >   Running mw0 tests on: 0001:10:00.0 /  0000:01:00.0
> >     Passed
> > 
> >   Starting ntb_pingpong tests...
> >   Running ping pong tests on: 0000:01:00.0 / 0001:10:00.0
> >     Passed
> > 
> >   Starting ntb_perf tests...
> >   Running local perf test without DMA
> >         Peer 0 test statistics:
> >   0: copied 268435456 bytes in 137063 usecs, 1958 MBytes/s
> >     Passed
> >   Running remote perf test without DMA
> >         Peer 0 test statistics:
> >   0: copied 268435456 bytes in 139188 usecs, 1928 MBytes/s
> >     Passed
> > 
> > For comparison, the before-series run reported 1997 MBytes/s locally
> > and 1950 MBytes/s remotely. No obvious regression was seen in this
> > single-run comparison.
> > 
> > 
> > Control layout compatibility
> > ============================
> > 
> > The endpoint selects the control layout. Non-packed pci-epf-vntb keeps
> > the version 0 layout. packed_mws selects version 1. "Old" and "new"
> > below mean ntb_hw_epf without and with this series.
> > 
> >   EP configuration  Old ntb_hw_epf            New ntb_hw_epf
> >   ----------------  ------------------------  ------------------------
> >   non-packed (v0)   Supported                 Supported
> >   packed (v1)       Not supported [1]         Supported
> > 
> > [1] The old driver accepts two or four MWs but looks for a separate BAR
> >     for each MW. It rejects larger MW counts at probe.
> > 
> > 
> > Packed MW compatibility with in-tree NTB clients
> > ================================================
> > 
> > The in-tree clients handle a packed MW group as follows:
> > 
> >   Client          Behavior
> >   --------------  ------------------------------------------------------
> >   ntb_transport   Uses group operations. Without a singleton MW, MSI
> >                   mode falls back to doorbells.
> >   ntb_perf        Rejects a non-singleton MW group at probe.
> >   ntb_tool        Loads normally. Per-MW inbound translation setup
> >                   returns -EOPNOTSUPP for a non-singleton group.
> >                   Doorbell and scratchpad access is unaffected.
> >   ntb_msi_test    Fails probe unless the MW reserved for MSI is a
> >                   singleton.
> >   ntb_pingpong    Does not use MWs and is unaffected.
> > 
> > Further client support can be added separately when needed. A final
> > singleton MW on another BAR can also be added later if MSI mode is needed
> > with packed MWs.
> > 
> > 
> > Notes
> > =====
> > 
> > * About pci-epf-ntb:
> > 
> >   pci-epf-ntb remains on the legacy version 0 layout; this series does
> >   not add version 1 layout support to it. A pci-epf-ntb bridge should
> >   therefore keep its existing behavior even when its nodes run a mix of
> >   old and new kernels. I do not have hardware for the pci-epf-ntb bridge
> >   topology.
> > 
> > * About existing inbound MW clear behavior:
> > 
> >   pci-epf-vntb already leaves the inbound BAR mapping in place when an
> >   MW translation is cleared. This series keeps that behavior for packed
> >   MWs. This limitation is independent of packing.
> > 
> >   pci_epc_clear_bar() is not suitable here because it can clear the BAR
> >   address assigned by the host. An EPF can instead restore a persistent
> >   fallback mapping with pci_epc_set_bar(), as pci-epf-test does:
> > 
> >     https://lore.kernel.org/r/20250908161942.534799-2-cassel@xxxxxxxxxx/
> > 
> >   pci-epf-vntb has no such fallback mapping today. Adding one, or an EPC
> >   operation to unmap only the inbound translation, should be separate
> >   work.
> > 
> > Best regards,
> > Koichiro
> > 
> > 
> > Koichiro Den (11):
> >   NTB: Add atomic MW translation group operations
> >   NTB: epf: Parse a versioned packed MW layout
> >   PCI: endpoint: pci-epf-vntb: Add packed MW layout handling
> >   PCI: endpoint: pci-epf-vntb: Implement MW group translation callbacks
> >   PCI: endpoint: pci-epf-vntb: Allocate packed outbound MW space
> >   PCI: endpoint: pci-epf-vntb: Add outbound MW group commands
> >   NTB: epf: Implement MW group translation callbacks
> >   NTB: perf: Reject grouped memory windows
> >   NTB/msi: Require a singleton memory window
> >   NTB: ntb_transport: Use atomic MW translation groups
> >   PCI: endpoint: pci-epf-vntb: Expose packed MWs through configfs
> > 
> >  Documentation/PCI/endpoint/pci-vntb-howto.rst |  16 +-
> >  drivers/ntb/hw/epf/ntb_hw_epf.c               | 143 +++++-
> >  drivers/ntb/msi.c                             |  19 +-
> >  drivers/ntb/ntb_transport.c                   | 127 +++++-
> >  drivers/ntb/test/ntb_perf.c                   |  16 +-
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 423 ++++++++++++++++--
> >  include/linux/ntb.h                           | 133 +++++-
> >  7 files changed, 801 insertions(+), 76 deletions(-)
> > 
> 
> -- 
> ~Randy
>