Re: [PATCH 00/11] PCI/NTB: endpoint: packed vNTB memory windows

[Randy Dunlap]

On 8/3/26 11:04 AM, Koichiro Den wrote:
> Hi,
> 
> vNTB currently spends one BAR per MW. BAR count therefore limits the
> number of MWs, even when one BAR has room for several. A large BAR,
> especially a Resizable BAR, can instead be split into logical MWs and
> leave more BARs for config and doorbells.
> 
> This series adds packed_mws, which exposes up to 16 equal-sized logical
> MWs in one BAR. ntb_transport uses the new group operations to configure
> them atomically.
> 
> The immediate use case is increasing the number of ntb_transport queues.
> The resulting performance gain is shown in the Testing section.
> 
> Based on the latest pci/endpoint.
> 
> 
> Why not BAR subrange mappings?
> ==============================
> 
> An earlier proposal used mwN_offset and BAR subrange mappings to place
> independently backed MWs in one BAR:
> 
>   https://lore.kernel.org/r/20260312165005.1148676-11-den@xxxxxxxxxxxxx/
> 
> BAR subrange mapping is currently implemented only by DWC, and that
> design still needs one hardware mapping per MW in each direction. This
> series instead requires one contiguous backing range and maps the whole
> group once in each direction. The initial implementation intentionally
> trades arbitrary MW placement for a constant mapping count as the MW
> count grows. More flexible layouts can be added when a concrete use
> case requires them.
> 

Hi,
To be clear, this is totally outside of the scope of any PCI or NTB
standards or specs?  or did I overlook that part of the discussion?
Thanks.

> 
> Testing
> =======
> 
> The packed data path was tested with ntb_transport and ntb_netdev. An
> `iperf3 -ub0 -l 65482 -P 2` UDP saturation run produced the following
> results when only one Resizable BAR was available for MWs:
> 
>   Configuration                                   Receiver throughput
>   ----------------------------------------------  -------------------
>   num_mws=1, packed_mws unset (default 0)         607 Mbits/sec
>   num_mws=2, packed_mws=2, combined channels=2    1.19 Gbits/sec
> 
> Meanwhile, I realized that the in-tree ntb_test.sh does not fully pass
> with the pci-epf-vntb/ntb_hw_epf setup on the base tree. I had to apply
> a separate collection of local PCI endpoint and NTB fixes before this
> setup could be used for regression testing. These fixes address
> pre-existing issues and are not prerequisites of the packed MW
> implementation.
> 
> With packed_mws unset and ntb_msi_tests disabled, ntb_test.sh reported
> the same pass/unsupported results before and after this series. One run
> follows:
> 
>   $ sudo ./ntb_test.sh -r ${EP} -m 1048576 -p 28 0000:01:00.0 0001:10:00.0
>   Starting ntb_tool tests...
>   Running port tests on: 0000:01:00.0 / 0001:10:00.0
>   Local port 0 with index 0 on remote host
>   Peer port 0 with index 0 on local host
>     Passed
>   Running link tests on: 0000:01:00.0 / 0001:10:00.0
>     Passed
>   Running link tests on: 0001:10:00.0 / 0000:01:00.0
>     Passed
>   Running db tests on: 0000:01:00.0 / 0001:10:00.0
>     Passed
>   Running db tests on: 0001:10:00.0 / 0000:01:00.0
>     Passed
>   Running spad tests on: 0000:01:00.0 / 0001:10:00.0
>     Passed
>   Running spad tests on: 0001:10:00.0 / 0000:01:00.0
>     Passed
>   Running msg tests on: 0000:01:00.0 / 0001:10:00.0
>     Unsupported
>   Running msg tests on: 0001:10:00.0 / 0000:01:00.0
>     Unsupported
>   Running mw0 tests on: 0000:01:00.0 /  0001:10:00.0
>     Passed
>   Running mw0 tests on: 0001:10:00.0 /  0000:01:00.0
>     Passed
> 
>   Starting ntb_pingpong tests...
>   Running ping pong tests on: 0000:01:00.0 / 0001:10:00.0
>     Passed
> 
>   Starting ntb_perf tests...
>   Running local perf test without DMA
>         Peer 0 test statistics:
>   0: copied 268435456 bytes in 137063 usecs, 1958 MBytes/s
>     Passed
>   Running remote perf test without DMA
>         Peer 0 test statistics:
>   0: copied 268435456 bytes in 139188 usecs, 1928 MBytes/s
>     Passed
> 
> For comparison, the before-series run reported 1997 MBytes/s locally
> and 1950 MBytes/s remotely. No obvious regression was seen in this
> single-run comparison.
> 
> 
> Control layout compatibility
> ============================
> 
> The endpoint selects the control layout. Non-packed pci-epf-vntb keeps
> the version 0 layout. packed_mws selects version 1. "Old" and "new"
> below mean ntb_hw_epf without and with this series.
> 
>   EP configuration  Old ntb_hw_epf            New ntb_hw_epf
>   ----------------  ------------------------  ------------------------
>   non-packed (v0)   Supported                 Supported
>   packed (v1)       Not supported [1]         Supported
> 
> [1] The old driver accepts two or four MWs but looks for a separate BAR
>     for each MW. It rejects larger MW counts at probe.
> 
> 
> Packed MW compatibility with in-tree NTB clients
> ================================================
> 
> The in-tree clients handle a packed MW group as follows:
> 
>   Client          Behavior
>   --------------  ------------------------------------------------------
>   ntb_transport   Uses group operations. Without a singleton MW, MSI
>                   mode falls back to doorbells.
>   ntb_perf        Rejects a non-singleton MW group at probe.
>   ntb_tool        Loads normally. Per-MW inbound translation setup
>                   returns -EOPNOTSUPP for a non-singleton group.
>                   Doorbell and scratchpad access is unaffected.
>   ntb_msi_test    Fails probe unless the MW reserved for MSI is a
>                   singleton.
>   ntb_pingpong    Does not use MWs and is unaffected.
> 
> Further client support can be added separately when needed. A final
> singleton MW on another BAR can also be added later if MSI mode is needed
> with packed MWs.
> 
> 
> Notes
> =====
> 
> * About pci-epf-ntb:
> 
>   pci-epf-ntb remains on the legacy version 0 layout; this series does
>   not add version 1 layout support to it. A pci-epf-ntb bridge should
>   therefore keep its existing behavior even when its nodes run a mix of
>   old and new kernels. I do not have hardware for the pci-epf-ntb bridge
>   topology.
> 
> * About existing inbound MW clear behavior:
> 
>   pci-epf-vntb already leaves the inbound BAR mapping in place when an
>   MW translation is cleared. This series keeps that behavior for packed
>   MWs. This limitation is independent of packing.
> 
>   pci_epc_clear_bar() is not suitable here because it can clear the BAR
>   address assigned by the host. An EPF can instead restore a persistent
>   fallback mapping with pci_epc_set_bar(), as pci-epf-test does:
> 
>     https://lore.kernel.org/r/20250908161942.534799-2-cassel@xxxxxxxxxx/
> 
>   pci-epf-vntb has no such fallback mapping today. Adding one, or an EPC
>   operation to unmap only the inbound translation, should be separate
>   work.
> 
> Best regards,
> Koichiro
> 
> 
> Koichiro Den (11):
>   NTB: Add atomic MW translation group operations
>   NTB: epf: Parse a versioned packed MW layout
>   PCI: endpoint: pci-epf-vntb: Add packed MW layout handling
>   PCI: endpoint: pci-epf-vntb: Implement MW group translation callbacks
>   PCI: endpoint: pci-epf-vntb: Allocate packed outbound MW space
>   PCI: endpoint: pci-epf-vntb: Add outbound MW group commands
>   NTB: epf: Implement MW group translation callbacks
>   NTB: perf: Reject grouped memory windows
>   NTB/msi: Require a singleton memory window
>   NTB: ntb_transport: Use atomic MW translation groups
>   PCI: endpoint: pci-epf-vntb: Expose packed MWs through configfs
> 
>  Documentation/PCI/endpoint/pci-vntb-howto.rst |  16 +-
>  drivers/ntb/hw/epf/ntb_hw_epf.c               | 143 +++++-
>  drivers/ntb/msi.c                             |  19 +-
>  drivers/ntb/ntb_transport.c                   | 127 +++++-
>  drivers/ntb/test/ntb_perf.c                   |  16 +-
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 423 ++++++++++++++++--
>  include/linux/ntb.h                           | 133 +++++-
>  7 files changed, 801 insertions(+), 76 deletions(-)
> 

-- 
~Randy