Re: [PATCH] PCI/PM: Put devices to low power state on shutdown

[Bjorn Helgaas]

[+cc Rafael]

On Thu, Sep 12, 2024 at 11:00:43AM +0800, Kai-Heng Feng wrote:
> On Thu, Sep 12, 2024 at 3:05 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> > On Fri, Jul 12, 2024 at 02:24:11PM +0800, Kai-Heng Feng wrote:
> > > Some laptops wake up after poweroff when HP Thunderbolt Dock G4 is
> > > connected.
> > >
> > > The following error message can be found during shutdown:
> > > pcieport 0000:00:1d.0: AER: Correctable error message received from 0000:09:04.0
> > > pcieport 0000:09:04.0: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Receiver ID)
> > > pcieport 0000:09:04.0:   device [8086:0b26] error status/mask=00000080/00002000
> > > pcieport 0000:09:04.0:    [ 7] BadDLLP
> > >
> > > Calling aer_remove() during shutdown can quiesce the error message,
> > > however the spurious wakeup still happens.
> > >
> > > The issue won't happen if the device is in D3 before system shutdown, so
> > > putting device to low power state before shutdown to solve the issue.
> > >
> > > I don't have a sniffer so this is purely guesswork, however I believe
> > > putting device to low power state it's the right thing to do.
> >
> > My objection here is that we don't have an explanation of why this
> > should matter or a pointer to any spec language about this situation,
> > so it feels a little bit random.
> 
> I have the same feeling too. The PCIe spec doesn't specify what's the
> correct power state for shutdown.
> So we can only "logically" think the software should put devices to
> low power state during shutdown.
> 
> > I suppose the problem wouldn't happen if AER interrupts were disabled?
> > We already do disable them in aer_suspend(), but maybe that's not used
> > in the shutdown path?
> 
> That was my first thought, so I modified pcie_port_shutdown_service()
> to disable AER interrupt.
> That approach didn't work though.
> 
> > My understanding is that .shutdown() should turn off device interrupts
> > and stop DMA.  So maybe we need an aer_shutdown() that disables
> > interrupts?
> 
> Logically we should do that. However that approach doesn't solve this issue.

I'm not completely clear on the semantics of the .shutdown()
interface.  The doc at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/device/driver.h?id=v6.10#n73
says "@shutdown: Called at shut-down time to quiesce the device"

Turning off device interrupts and DMA *would* fit within the idea of
quiescing the device.  Does that also include changing the device
power state?  I dunno.  The power state isn't *mentioned* in the
.shutdown() context, while it *is* mentioned for .suspend().

IIUC, this patch and commit log uses "shutdown" to refer to a
system-wide *poweroff*, which is a different concept despite using the
same "shutdown" name.

So should the system poweroff procedure use .suspend()?  Should it use
both .shutdown() and .suspend()?  I think it only uses .shutdown()
today:

  kernel_power_off
    kernel_shutdown_prepare(SYSTEM_POWER_OFF)
      device_shutdown
        while (!list_empty(&devices_kset->list))
          dev->bus->shutdown(dev)
            pci_device_shutdown

There are several driver .shutdown() methods that do things like this:

  e1000_shutdown
    if (system_state == SYSTEM_POWER_OFF)
      pci_set_power_state(pdev, PCI_D3hot)

Maybe that's the right thing and should be done by the PCI core, which
is similar to what you propose here.  But I think it muddies the
definition of .shutdown() a bit by mixing in power management stuff.

> > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=219036
> > > Signed-off-by: Kai-Heng Feng <kai.heng.feng@xxxxxxxxxxxxx>
> > > ---
> > >  drivers/pci/pci-driver.c | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > >
> > > diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
> > > index af2996d0d17f..4c6f66f3eb54 100644
> > > --- a/drivers/pci/pci-driver.c
> > > +++ b/drivers/pci/pci-driver.c
> > > @@ -510,6 +510,14 @@ static void pci_device_shutdown(struct device *dev)
> > >       if (drv && drv->shutdown)
> > >               drv->shutdown(pci_dev);
> > >
> > > +     /*
> > > +      * If driver already changed device's power state, it can mean the
> > > +      * wakeup setting is in place, or a workaround is used. Hence keep it
> > > +      * as is.
> > > +      */
> > > +     if (!kexec_in_progress && pci_dev->current_state == PCI_D0)
> > > +             pci_prepare_to_sleep(pci_dev);
> > > +
> > >       /*
> > >        * If this is a kexec reboot, turn off Bus Master bit on the
> > >        * device to tell it to not continue to do DMA. Don't touch
> > > --
> > > 2.43.0
> > >