[PATCH 0/2] PCI/AER: Fix and document aer_inject's aer_mask_override

[Priyank Rathod]

The aer_mask_override parameter of aer_inject does not work for
Correctable or Non-Fatal injections: the hardware Mask registers are
restored before the threaded AER handler reads them, so
aer_get_device_error_info() drops the injected error as masked.  The
only injections for which the parameter ever produced a handled error
were Fatal errors on device types whose AER registers
aer_get_device_error_info() doesn't read (e.g., Endpoints or Upstream
Ports), and even there only because the -EINVAL pre-check was skipped,
not because of the mask writes.  The '!' vs '~' operator bug in the
same code is a separate defect that transiently unmasks every error of
the device.

Why fix rather than remove the parameter: it dates from 2011
(457d9d088b02 ("PCI: aer-inject: Override PCIe AER Mask Registers"))
and is mentioned in the RHEL 6.2 Technical Notes, so test scripts may
still rely on it; it is also the only way to exercise handling of
errors that are masked by default (e.g., Header Log Overflow or
Poisoned TLP Egress Blocked, or the Internal Errors on non-CXL
devices) without reprogramming the device via setpci, which the
kernel's own pci_restore_aer_state() may later undo.

Patch 1 stops touching the hardware Mask registers, emulates the
override in config space reads while the injected error is pending in
the emulated Status registers, like the status emulation itself, and
keeps the emulated value from being written back to hardware by
pci_restore_state().  The '!' vs '~' fix is folded in because on its
own it has no observable effect on the injection outcome (the masks
were restored before the interrupt anyway) and the lines are removed
by the emulation.

Patch 2 documents the parameter in the AER HOWTO, which never
mentioned it.

What the mask writes in aer_inject() do today, with the register
values of the sample session below (PCI_ERR_COR_MASK 0000c040 with
BadTLP masked, PCI_ERR_UNCOR_MASK 02400000 untouched, COR_STATUS
BAD_TLP = 00000040 injected):

  intended:  cor_mask   &= ~00000040   0000c040 -> 0000c000
             uncor_mask &= ~00000000   02400000 -> 02400000
  actual:    cor_mask   &= !00000040   0000c040 & 0 -> 00000000
             uncor_mask &= !00000000   02400000 & 1 -> 00000000

So between the write and the restore every Correctable and every
Uncorrectable error of the device is unmasked, including the ones it
masks by default (CorrIntErr, HeaderOF, UncorrIntErr, ...).  Both
registers are restored to 0000c040/02400000 before
irq_inject_interrupt() is called, so when the threaded AER handler
reads them through the emulated config space it gets the hardware
value and evaluates

  status & ~mask == 00000040 & ~0000c040 == 0

i.e. it drops the injected error as masked and the emulated status bit
is left pending.  With the patch the hardware is never written and the
override is applied where it is consumed, in the read:

  aer_isr() reads PCI_ERR_COR_MASK while 00000040 is pending in the
  emulated COR_STATUS:  0000c040 & ~00000040 -> 0000c000, so
  status & ~mask == 00000040 and "[ 6] BadTLP" is handled.
  Any read after the status was cleared, or after rmmod: 0000c040.

The write side, for a pending Unsupported Request with
PCI_ERR_UNCOR_MASK 02500000 (UR masked):

  pci_save_state()     reads 02500000 & ~00100000 -> 02400000
  pci_restore_state()  writes 02400000; aer_inject sets the hidden bit
                       again: 02400000 | 00100000 -> 02500000

Sample session in a QEMU q35 VM (pcie-root-port + e1000e at 01:00.0,
pcie_ports=native, v7.3-rc1 with aer_inject built before and after the
patch, injections done with the aer-inject tool):

  # setpci -s 01:00.0 ECAP_AER+0x14.L
  0000c000
  # setpci -s 01:00.0 ECAP_AER+0x14.L=0000c040        (mask BadTLP)
  # insmod aer_inject.ko aer_mask_override=1
  # aer-inject -s 01:00.0 cor.conf                    (COR_STATUS BAD_TLP)

Before:

  pcieport 0000:00:01.0: AER: Correctable Error message received from 0000:01:00.0
  # grep BadTLP /sys/bus/pci/devices/0000:01:00.0/aer_dev_correctable
  BadTLP 0
  # setpci -s 01:00.0 ECAP_AER+0x10.L                 (emulated COR_STATUS)
  00000040                                            (left pending)

After:

  pcieport 0000:00:01.0: AER: Correctable Error message received from 0000:01:00.0
  e1000e 0000:01:00.0: PCIe Bus Error: severity=Correctable
  e1000e 0000:01:00.0:   device [8086:10d3] error status/mask=00000040/0000c000
  e1000e 0000:01:00.0:   [ 6] BadTLP            | Receiver    | Data Link Layer
  # grep BadTLP /sys/bus/pci/devices/0000:01:00.0/aer_dev_correctable
  BadTLP 1
  # setpci -s 01:00.0 ECAP_AER+0x10.L
  00000000
  # rmmod aer_inject; setpci -s 01:00.0 ECAP_AER+0x14.L
  0000c040                                            (hardware untouched)

The same with a masked Unsupported Request (UNCOR_STATUS UNSUP,
PCI_ERR_UNCOR_MASK 02500000): before, only "Uncorrectable (Non-Fatal)
Error message received", UnsupReq count 0 and the emulated status
00100000 left pending; after, "[20] UnsupReq" is decoded, the count is
1 and pcie_do_recovery() runs.  Recovery fails in this VM ("Data Link
Layer Link Active not set in 100 msec"), which leaves the injected
error pending for the last test.  With aer_mask_override=0 the
injection is still refused: "The correctable error(s) is masked by
device".

Reset while the injected UR is still pending, i.e. pci_save_state()
reads the Mask register through the emulation and pci_restore_state()
writes it back:

  # setpci -s 01:00.0 ECAP_AER+0x08.L           (emulated: UR unmasked)
  02400000
  # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
  # rmmod aer_inject; setpci -s 01:00.0 ECAP_AER+0x08.L
  02500000                                            (hardware untouched)

With the read-side emulation alone (an earlier version of this patch)
the last read returned 02400000: pci_restore_state() had written the
emulated value to the device and UR stayed unmasked in hardware after
the unload.  That is why the patch also keeps the hidden Mask bits set
in writes to the Mask registers.

Signed-off-by: Priyank Rathod <rathodpriyank@xxxxxxxxxx>
---
Priyank Rathod (2):
      PCI/AER: Make aer_mask_override effective for injected errors
      Documentation: PCI: Document aer_inject's aer_mask_override parameter

 Documentation/PCI/pcieaer-howto.rst | 23 ++++++++++++
 drivers/pci/pcie/aer_inject.c       | 73 +++++++++++++++++++++++++------------
 2 files changed, 73 insertions(+), 23 deletions(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20261012-fix-aer-mask-override-6b7d416c3d7c

Best regards,
-- 
Priyank Rathod <rathodpriyank@xxxxxxxxxx>