RE: [PATCH v2] rtw88: Disable PCIe ASPM while doing NAPI poll on 8821CE

[Pkshih]

> -----Original Message-----
> From: Kai-Heng Feng <kai.heng.feng@xxxxxxxxxxxxx>
> Sent: Tuesday, December 14, 2021 2:07 PM
> To: Pkshih <pkshih@xxxxxxxxxxx>
> Cc: tony0620emma@xxxxxxxxx; jian-hong@xxxxxxxxxxxx <jhp@xxxxxxxxxxxxx>; Kalle Valo <kvalo@xxxxxxxxxxxxxx>;
> David S. Miller <davem@xxxxxxxxxxxxx>; Jakub Kicinski <kuba@xxxxxxxxxx>; Bernie Huang
> <phhuang@xxxxxxxxxxx>; Brian Norris <briannorris@xxxxxxxxxxxx>; linux-wireless@xxxxxxxxxxxxxxx;
> netdev@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH v2] rtw88: Disable PCIe ASPM while doing NAPI poll on 8821CE
> 
> On Tue, Dec 14, 2021 at 1:59 PM Pkshih <pkshih@xxxxxxxxxxx> wrote:
> >
> >
> > > -----Original Message-----
> > > From: Kai-Heng Feng <kai.heng.feng@xxxxxxxxxxxxx>
> > > Sent: Tuesday, December 14, 2021 1:33 PM
> > > To: tony0620emma@xxxxxxxxx; Pkshih <pkshih@xxxxxxxxxxx>
> > > Cc: jian-hong@xxxxxxxxxxxx; Kai-Heng Feng <kai.heng.feng@xxxxxxxxxxxxx>; Kalle Valo
> > > <kvalo@xxxxxxxxxxxxxx>; David S. Miller <davem@xxxxxxxxxxxxx>; Jakub Kicinski <kuba@xxxxxxxxxx>; Bernie
> > > Huang <phhuang@xxxxxxxxxxx>; Brian Norris <briannorris@xxxxxxxxxxxx>; linux-wireless@xxxxxxxxxxxxxxx;
> > > netdev@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx
> > > Subject: [PATCH v2] rtw88: Disable PCIe ASPM while doing NAPI poll on 8821CE
> > >
> > > Many Intel based platforms face system random freeze after commit
> > > 9e2fd29864c5 ("rtw88: add napi support").
> > >
> > > The commit itself shouldn't be the culprit. My guess is that the 8821CE
> > > only leaves ASPM L1 for a short period when IRQ is raised. Since IRQ is
> > > masked during NAPI polling, the PCIe link stays at L1 and makes RX DMA
> > > extremely slow. Eventually the RX ring becomes messed up:
> > > [ 1133.194697] rtw_8821ce 0000:02:00.0: pci bus timeout, check dma status
> > >
> > > Since the 8821CE hardware may fail to leave ASPM L1, manually do it in
> > > the driver to resolve the issue.
> > >
> > > Fixes: 9e2fd29864c5 ("rtw88: add napi support")
> > > Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=215131
> > > BugLink: https://bugs.launchpad.net/bugs/1927808
> > > Signed-off-by: Kai-Heng Feng <kai.heng.feng@xxxxxxxxxxxxx>
> > > ---
> > > v2:
> > >  - Add default value for module parameter.
> > >
> > >  drivers/net/wireless/realtek/rtw88/pci.c | 74 ++++++++----------------
> > >  drivers/net/wireless/realtek/rtw88/pci.h |  1 +
> > >  2 files changed, 24 insertions(+), 51 deletions(-)
> > >
> > > diff --git a/drivers/net/wireless/realtek/rtw88/pci.c b/drivers/net/wireless/realtek/rtw88/pci.c
> > > index 3b367c9085eba..4ab75ac2500e9 100644
> > > --- a/drivers/net/wireless/realtek/rtw88/pci.c
> > > +++ b/drivers/net/wireless/realtek/rtw88/pci.c
> > > @@ -2,7 +2,6 @@
> > >  /* Copyright(c) 2018-2019  Realtek Corporation
> > >   */
> > >
> > > -#include <linux/dmi.h>
> > >  #include <linux/module.h>
> > >  #include <linux/pci.h>
> > >  #include "main.h"
> > > @@ -16,10 +15,13 @@
> > >
> > >  static bool rtw_disable_msi;
> > >  static bool rtw_pci_disable_aspm;
> > > +static int rtw_rx_aspm = -1;
> > >  module_param_named(disable_msi, rtw_disable_msi, bool, 0644);
> > >  module_param_named(disable_aspm, rtw_pci_disable_aspm, bool, 0644);
> > > +module_param_named(rx_aspm, rtw_rx_aspm, int, 0444);
> > >  MODULE_PARM_DESC(disable_msi, "Set Y to disable MSI interrupt support");
> > >  MODULE_PARM_DESC(disable_aspm, "Set Y to disable PCI ASPM support");
> > > +MODULE_PARM_DESC(rx_aspm, "Use PCIe ASPM for RX (0=disable, 1=enable, -1=default)");
> > >
> > >  static u32 rtw_pci_tx_queue_idx_addr[] = {
> > >       [RTW_TX_QUEUE_BK]       = RTK_PCI_TXBD_IDX_BKQ,
> > > @@ -1409,7 +1411,11 @@ static void rtw_pci_link_ps(struct rtw_dev *rtwdev, bool enter)
> > >        * throughput. This is probably because the ASPM behavior slightly
> > >        * varies from different SOC.
> > >        */
> > > -     if (rtwpci->link_ctrl & PCI_EXP_LNKCTL_ASPM_L1)
> > > +     if (!(rtwpci->link_ctrl & PCI_EXP_LNKCTL_ASPM_L1))
> > > +             return;
> > > +
> > > +     if ((enter && atomic_dec_return(&rtwpci->link_usage) == 0) ||
> > > +         (!enter && atomic_inc_return(&rtwpci->link_usage) == 1))
> > >               rtw_pci_aspm_set(rtwdev, enter);
> > >  }
> > >
> >
> > I found calling pci_link_ps isn't always symmetric, so we need to reset
> > ref_cnt at pci_start() like below, or we can't enter rtw_pci_aspm_set()
> > anymore. The negative flow I face is
> > ifup -> connect AP -> ifdown -> ifup (ref_cnt isn't expected now).
> 
> Is it expected to be asymmetric?
> If it's intended to be this way, I'll change where we set link_usage.
> Otherwise I think making it symmetric makes more sense.

Agree with your thought, but I don't remember clearly why it enters idle twice
in above flow. So, you may use two flags to indicate the state wanted by
two callers.

--
Ping-Ke