Re: [PATCH v7 0/5] Fix DPC hotplug race and enhance error handling

[Ethan Zhao]

Raj,

On Sun, Oct 4, 2020 at 12:57 PM Raj, Ashok <ashok.raj@xxxxxxxxxxxxxxx> wrote:
>
> Hi Ethan
>
> On Sat, Oct 03, 2020 at 03:55:09AM -0400, Ethan Zhao wrote:
> > Hi,folks,
> >
> > This simple patch set fixed some serious security issues found when DPC
> > error injection and NVMe SSD hotplug brute force test were doing -- race
> > condition between DPC handler and pciehp, AER interrupt handlers, caused
> > system hang and system with DPC feature couldn't recover to normal
> > working state as expected (NVMe instance lost, mount operation hang,
> > race PCIe access caused uncorrectable errors reported alternatively etc).
>
> I think maybe picking from other commit messages to make this description in
> cover letter bit clear. The fundamental premise is that when due to error
> conditions when events are processed by both DPC handler and hotplug handling of
> DLLSC both operating on the same device object ends up with crashes.

Yep, that's right.

Thanks,
Ethan
>
>
> >
> > With this patch set applied, stable 5.9-rc6 on ICS (Ice Lake SP platform,
> > see
> > https://en.wikichip.org/wiki/intel/microarchitectures/ice_lake_(server))
> >
> > could pass the PCIe Gen4 NVMe SSD brute force hotplug test with any time
> > interval between hot-remove and plug-in operation tens of times without
> > any errors occur and system works normal.
>
> >
> > With this patch set applied, system with DPC feature could recover from
> > NON-FATAL and FATAL errors injection test and works as expected.
> >
> > System works smoothly when errors happen while hotplug is doing, no
> > uncorrectable errors found.
> >
> > Brute DPC error injection script:
> >
> > for i in {0..100}
> > do
> >         setpci -s 64:02.0 0x196.w=000a
> >         setpci -s 65:00.0 0x04.w=0544
> >         mount /dev/nvme0n1p1 /root/nvme
> >         sleep 1
> > done
> >
> > Other details see every commits description part.
> >
> > This patch set could be applied to stable 5.9-rc6/rc7 directly.
> >
> > Help to review and test.
> >
> > v2: changed according to review by Andy Shevchenko.
> > v3: changed patch 4/5 to simpler coding.
> > v4: move function pci_wait_port_outdpc() to DPC driver and its
> >    declaration to pci.h. (tip from Christoph Hellwig <hch@xxxxxxxxxxxxx>).
> > v5: fix building issue reported by lkp@xxxxxxxxx with some config.
> > v6: move patch[3/5] as the first patch according to Lukas's suggestion.
> >     and rewrite the comment part of patch[3/5].
> > v7: change the patch[4/5], based on Bjorn's code and truth table.
> >     change the patch[5/5] about the debug output information.
> >
> > Thanks,
> > Ethan
> >
> >
> > Ethan Zhao (5):
> >   PCI/ERR: get device before call device driver to avoid NULL pointer
> >     dereference
> >   PCI/DPC: define a function to check and wait till port finish DPC
> >     handling
> >   PCI: pciehp: check and wait port status out of DPC before handling
> >     DLLSC and PDC
> >   PCI: only return true when dev io state is really changed
> >   PCI/ERR: don't mix io state not changed and no driver together
> >
> >  drivers/pci/hotplug/pciehp_hpc.c |  4 ++-
> >  drivers/pci/pci.h                | 55 +++++++++++++-------------------
> >  drivers/pci/pcie/dpc.c           | 27 ++++++++++++++++
> >  drivers/pci/pcie/err.c           | 18 +++++++++--
> >  4 files changed, 68 insertions(+), 36 deletions(-)
> >
> >
> > base-commit: a1b8638ba1320e6684aa98233c15255eb803fac7
> > --
> > 2.18.4
> >