[PATCH 2/2] iommu/amd: Register PCI bus notifier when appending amd_iommu=off

[Adrian Huang (Lenovo)]

When the amd_iommu=off kernel parameter is used, dynamically adding
PCI devices (e.g., creating SR-IOV VFs) triggers a call trace and
IOMMU fault events. For example, enabling VFs via sysfs:

  # echo 4 > /sys/class/net/eth1/device/sriov_numvfs

It results in an INVALID_DEVICE_REQUEST and a warning in
__irq_msi_compose_msg+0xa3/0xb0:

  pci 0000:41:03.0: [14e4:16c1] type 00 class 0x020000 PCIe Endpoint
  bnxt_en 0000:41:03.0: enabling device (0000 -> 0002)
  bnxt_en 0000:41:03.0 eth4: Broadcom NetXtreme-E Ethernet Virtual Function found at mem 381c7ff40000, node addr 92:42:39:64:ad:24
  ...
  WARNING: arch/x86/kernel/apic/apic.c:2313 at __irq_msi_compose_msg+0xa3/0xb0, CPU#2: NetworkManager/4268
  bnxt_en 0000:41:03.1: enabling device (0000 -> 0002)
  CPU: 2 UID: 0 PID: 4268 Comm: NetworkManager Kdump: loaded Not tainted 7.0.0-rc4+ #23 PREEMPT(lazy)
  ...
  RIP: 0010:__irq_msi_compose_msg+0xa3/0xb0
  ...
  Call Trace:
   <TASK>
   irq_chip_compose_msi_msg+0x2e/0x50
   msi_domain_activate+0x41/0x90
   __irq_domain_activate_irq+0x53/0x90
   ? mtree_load+0x26e/0x2b0
   irq_domain_activate_irq+0x29/0x40
   __setup_irq+0x322/0x790
   ? __pfx_bnxt_msix+0x10/0x10 [bnxt_en]
   request_threaded_irq+0x109/0x1c0
   bnxt_request_irq+0xee/0x270 [bnxt_en]
   ...
   AMD-Vi: Event logged [INVALID_DEVICE_REQUEST device=0000:41:03.0 pasid=0x00000 address=0xfffffffdf8220100 flags=0x0a00]

The root cause is that when IOMMU is disabled via the command line, the
PCI bus notifier is not registered. Consequently, the MSI domain for
dynamically added devices is not properly associated with the IOMMU
interrupt remapping domain. Even when the IOMMU is "off," certain
interrupt remapping configurations must still be handled for newly
enumerated devices to prevent activation failures.

Fix this by ensuring the PCI bus notifier is registered even when
amd_iommu=off is set, allowing the IOMMU interrupt remapping domain to
be correctly assigned to new PCI devices.

Signed-off-by: Adrian Huang (Lenovo) <adrianhuang0701@xxxxxxxxx>
---
 drivers/iommu/amd/init.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 0f577534702d..5509ade0cf0d 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -3431,6 +3431,30 @@ static int __init amd_iommu_devices_set_pci_msi_domain(void)
 	return ret;
 }
 
+static int amd_iommu_pci_bus_notifier(struct notifier_block *nb,
+					unsigned long action, void *data)
+{
+	struct pci_dev *pdev = to_pci_dev(data);
+
+	/* We only care about the add event. */
+	if (action != BUS_NOTIFY_ADD_DEVICE)
+		return NOTIFY_DONE;
+
+	amd_iommu_dev_set_pci_msi_domain(&pdev->dev);
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block amd_iommu_pci_bus_nb = {
+	.notifier_call = amd_iommu_pci_bus_notifier,
+	.priority = 1,
+};
+
+static void __init amd_iommu_register_bus_notifier(void)
+{
+	bus_register_notifier(&pci_bus_type, &amd_iommu_pci_bus_nb);
+}
+
 /****************************************************************************
  *
  * AMD IOMMU Initialization State Machine
@@ -3462,6 +3486,7 @@ static int __init state_next(void)
 	case IOMMU_ENABLED:
 		if (amd_iommu_disabled) {
 			amd_iommu_devices_set_pci_msi_domain();
+			amd_iommu_register_bus_notifier();
 			init_state = IOMMU_CMDLINE_DISABLED;
 			ret = -EINVAL;
 		} else {
-- 
2.47.3