Re: [PATCH V4 00/12] PCIe TPH and cache direct injection support

[Wei Huang]

Hi Bjorn,

I have incorporated Jakub's feedback on patch 11 (i.e. bnxt.c) in my new 
revision. Do you have additional comments or suggestions on V4 to 
include in the next spin?

Thanks,
-Wei

On 8/22/24 3:41 PM, Wei Huang wrote:
> Hi All,
>
> TPH (TLP Processing Hints) is a PCIe feature that allows endpoint
> devices to provide optimization hints for requests that target memory
> space. These hints, in a format called steering tag (ST), are provided
> in the requester's TLP headers and allow the system hardware, including
> the Root Complex, to optimize the utilization of platform resources
> for the requests.
>
> Upcoming AMD hardware implement a new Cache Injection feature that
> leverages TPH. Cache Injection allows PCIe endpoints to inject I/O
> Coherent DMA writes directly into an L2 within the CCX (core complex)
> closest to the CPU core that will consume it. This technology is aimed
> at applications requiring high performance and low latency, such as
> networking and storage applications.
>
> This series introduces generic TPH support in Linux, allowing STs to be
> retrieved and used by PCIe endpoint drivers as needed. As a
> demonstration, it includes an example usage in the Broadcom BNXT driver.
> When running on Broadcom NICs with the appropriate firmware, it shows
> substantial memory bandwidth savings and better network bandwidth using
> real-world benchmarks. This solution is vendor-neutral and implemented
> based on industry standards (PCIe Spec and PCI FW Spec).
>
> V3->V4:
>   * Rebase on top of the latest pci/next tree (tag: 6.11-rc1)
>   * Add new API functioins to query/enable/disable TPH support
>   * Make pcie_tph_set_st() completely independent from pcie_tph_get_cpu_st()
>   * Rewrite bnxt.c based on new APIs
>   * Remove documentation for now due to constantly changing API
>   * Remove pci=notph, but keep pci=nostmode with better flow (Bjorn)
>   * Lots of code rewrite in tph.c & pci-tph.h with cleaner interface (Bjorn)
>   * Add TPH save/restore support (Paul Luse and Lukas Wunner)
>
> V2->V3:
>   * Rebase on top of pci/next tree (tag: pci-v6.11-changes)
>   * Redefine PCI TPH registers (pci_regs.h) without breaking uapi
>   * Fix commit subjects/messages for kernel options (Jonathan and Bjorn)
>   * Break API functions into three individual patches for easy review
>   * Rewrite lots of code in tph.c/tph.h based (Jonathan and Bjorn)
>
> V1->V2:
>   * Rebase on top of pci.git/for-linus (6.10-rc1)
>   * Address mismatched data types reported by Sparse (Sparse check passed)
>   * Add pcie_tph_intr_vec_supported() for checking IRQ mode support
>   * Skip bnxt affinity notifier registration if
>     pcie_tph_intr_vec_supported()=false
>   * Minor fixes in bnxt driver (i.e. warning messages)
>
> Manoj Panicker (1):
>    bnxt_en: Add TPH support in BNXT driver
>
> Michael Chan (1):
>    bnxt_en: Pass NQ ID to the FW when allocating RX/RX AGG rings
>
> Paul Luse (1):
>    PCI/TPH: Add save/restore support for TPH
>
> Wei Huang (9):
>    PCI: Introduce PCIe TPH support framework
>    PCI: Add TPH related register definition
>    PCI/TPH: Add pcie_tph_modes() to query TPH modes
>    PCI/TPH: Add pcie_enable_tph() to enable TPH
>    PCI/TPH: Add pcie_disable_tph() to disable TPH
>    PCI/TPH: Add pcie_tph_enabled() to check TPH state
>    PCI/TPH: Add pcie_tph_set_st_entry() to set ST tag
>    PCI/TPH: Add pcie_tph_get_cpu_st() to get ST tag
>    PCI/TPH: Add pci=nostmode to force TPH No ST Mode
>
>   .../admin-guide/kernel-parameters.txt         |   3 +
>   drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  86 ++-
>   drivers/net/ethernet/broadcom/bnxt/bnxt.h     |   4 +
>   drivers/pci/pci.c                             |   4 +
>   drivers/pci/pci.h                             |  12 +
>   drivers/pci/pcie/Kconfig                      |  11 +
>   drivers/pci/pcie/Makefile                     |   1 +
>   drivers/pci/pcie/tph.c                        | 563 ++++++++++++++++++
>   drivers/pci/probe.c                           |   1 +
>   include/linux/pci-tph.h                       |  48 ++
>   include/linux/pci.h                           |   7 +
>   include/uapi/linux/pci_regs.h                 |  38 +-
>   12 files changed, 768 insertions(+), 10 deletions(-)
>   create mode 100644 drivers/pci/pcie/tph.c
>   create mode 100644 include/linux/pci-tph.h