Re: [PATCH v9 0/2] PCI: add enabe(disable)_device() hook for bridge

[Bjorn Helgaas]

On Tue, Jan 14, 2025 at 03:37:07PM -0500, Frank Li wrote:
> Some system's IOMMU stream(master) ID bits(such as 6bits) less than
> pci_device_id (16bit). It needs add hardware configuration to enable
> pci_device_id to stream ID convert.
> 
> https://lore.kernel.org/imx/20240622173849.GA1432357@bhelgaas/
> This ways use pcie bus notifier (like apple pci controller), when new PCIe
> device added, bus notifier will call register specific callback to handle
> look up table (LUT) configuration.
> 
> https://lore.kernel.org/imx/20240429150842.GC1709920-robh@xxxxxxxxxx/
> which parse dt's 'msi-map' and 'iommu-map' property to static config LUT
> table (qcom use this way). This way is rejected by DT maintainer Rob.
> 
> Above ways can resolve LUT take or stream id out of usage the problem. If
> there are not enough stream id resource, not error return, EP hardware
> still issue DMA to do transfer, which may transfer to wrong possition.
> 
> Add enable(disable)_device() hook for bridge can return error when not
> enough resource, and PCI device can't enabled.
> 
> Basicallly this version can match Bjorn's requirement:
> 1: simple, because it is rare that there are no LUT resource.
> 2: EP driver probe failure when no LUT, but lspci can see such device.
> 
> [    2.164415] nvme nvme0: pci function 0000:01:00.0
> [    2.169142] pci 0000:00:00.0: Error enabling bridge (-1), continuing
> [    2.175654] nvme 0000:01:00.0: probe with driver nvme failed with error -12
> 
> > lspci
> 0000:00:00.0 PCI bridge: Philips Semiconductors Device 0000
> 0000:01:00.0 Non-Volatile memory controller: Micron Technology Inc 2100AI NVMe SSD [Nitro] (rev 03)
> 
> To: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
> To: Richard Zhu <hongxing.zhu@xxxxxxx>
> To: Lucas Stach <l.stach@xxxxxxxxxxxxxx>
> To: Lorenzo Pieralisi <lpieralisi@xxxxxxxxxx>
> To: Krzysztof Wilczyński <kw@xxxxxxxxx>
> To: Manivannan Sadhasivam <manivannan.sadhasivam@xxxxxxxxxx>
> To: Rob Herring <robh@xxxxxxxxxx>
> To: Shawn Guo <shawnguo@xxxxxxxxxx>
> To: Sascha Hauer <s.hauer@xxxxxxxxxxxxxx>
> To: Pengutronix Kernel Team <kernel@xxxxxxxxxxxxxx>
> To: Fabio Estevam <festevam@xxxxxxxxx>
> Cc: linux-pci@xxxxxxxxxxxxxxx
> Cc: linux-kernel@xxxxxxxxxxxxxxx
> Cc: linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> Cc: imx@xxxxxxxxxxxxxxx
> Cc: Frank.li@xxxxxxx \
> Cc: alyssa@xxxxxxxxxxxxx \
> Cc: bpf@xxxxxxxxxxxxxxx \
> Cc: broonie@xxxxxxxxxx \
> Cc: jgg@xxxxxxxx \
> Cc: joro@xxxxxxxxxx \
> Cc: l.stach@xxxxxxxxxxxxxx \
> Cc: lgirdwood@xxxxxxxxx \
> Cc: maz@xxxxxxxxxx \
> Cc: p.zabel@xxxxxxxxxxxxxx \
> Cc: robin.murphy@xxxxxxx \
> Cc: will@xxxxxxxxxx \
> Cc: Robin Murphy <robin.murphy@xxxxxxx>
> Cc: Marc Zyngier <maz@xxxxxxxxxx>
> 
> Signed-off-by: Frank Li <Frank.Li@xxxxxxx>

Applied to pci/controller/imx6 for v6.14, thanks!  And thanks for your
patience.

> ---
> Changes in v9:
> - update commit message and comments
> - Rob agree use API to parse iommu-map and msi-map.
> https://lore.kernel.org/imx/20250113225905.GA3325507-robh@xxxxxxxxxx/
> - Link to v8: https://lore.kernel.org/r/20241210-imx95_lut-v8-0-2e730b2e5fde@xxxxxxx
> 
> Changes in v8:
> - update comment message according to Lorenzo Pieralisi's suggestion.
> - rework err target table
> - improve err==0 && target ==NULL description, use 1:1 map RID to
> stream ID.
> - invalidate case -> unexisted case, never happen
> - sid_i will not do mask, add comments said only MSI glue layer add
> controller id.
> - rework iommu map and msi map return value check logic according to
> Lorenzo Pieralisi's suggestion
> - Link to v7: https://lore.kernel.org/r/20241203-imx95_lut-v7-0-d0cd6293225e@xxxxxxx
> 
> Changes in v7:
> - Rebase v6.13-rc1
> - Update patch 2 according to mani's feedback
> - Link to v6: https://lore.kernel.org/r/20241118-imx95_lut-v6-0-a2951ba13347@xxxxxxx
> 
> Changes in v6:
> - Bjorn give review tags at v4, but v5 have big change, drop Bjorn's review
> tag.
> - Add back Marc Zyngier't review and test tags
> - Add mani's ack at first patch
> - Mini change for patch 2 according to mani's feedback
> - Link to v5: https://lore.kernel.org/r/20241104-imx95_lut-v5-0-feb972f3f13b@xxxxxxx
> 
> Changes in v5:
> - Add help function of pci_bridge_enable(disable)_device
> - Because big change, removed Bjorn's review tags and have not
> added
> Marc Zyngier't review and test tags
> - Fix pci-imx6.c according to Mani's feedback
> - Link to v4: https://lore.kernel.org/r/20241101-imx95_lut-v4-0-0fdf9a2fe754@xxxxxxx
> 
> Changes in v4:
> - Add Bjorn Helgaas review tag for patch1
> - check 'target' value for patch2
> - detail see each patches
> - Link to v3: https://lore.kernel.org/r/20241024-imx95_lut-v3-0-7509c9bbab86@xxxxxxx
> 
> Changes in v3:
> - disable_device when error happen
> - use target for of_map_id
> - Check if rid already in lut table when enable deviced
> - Link to v2: https://lore.kernel.org/r/20240930-imx95_lut-v2-0-3b6467ba539a@xxxxxxx
> 
> Changes in v2:
> - see each patch
> - Link to v1: https://lore.kernel.org/r/20240926-imx95_lut-v1-0-d0c62087dbab@xxxxxxx
> 
> ---
> Frank Li (2):
>       PCI: Add enable_device() and disable_device() callbacks for bridges
>       PCI: imx6: Add IOMMU and ITS MSI support for i.MX95
> 
>  drivers/pci/controller/dwc/pci-imx6.c | 199 +++++++++++++++++++++++++++++++++-
>  drivers/pci/pci.c                     |  36 +++++-
>  include/linux/pci.h                   |   2 +
>  3 files changed, 235 insertions(+), 2 deletions(-)
> ---
> base-commit: 40384c840ea1944d7c5a392e8975ed088ecf0b37
> change-id: 20240926-imx95_lut-1c68222e0944
> 
> Best regards,
> ---
> Frank Li <Frank.Li@xxxxxxx>
>