Re: [PATCH ath-next v2 0/5] wifi: ath11k: firmware crash recovery fixes

[Florian Maurer]

Hi Julius,

I have been chasing the same recovery problems on IPQ8074 (Xiaomi
AX3600, WLAN.HK.2.9.0.1-02146) in a Freifunk mesh network and ended up
with fixes very close to your 2/5-5/5, so I tested your series instead.

Setup: OpenWrt 25.12, kernel 6.12.112, your series backported to
backports-6.18.55 on top of the three ath-next IRQ fixes (a7ab5c835e42,
f7a74e131d3f, e56c9d429ba0), plus two fixes of mine that I posted on
top of your series [1], together with a Kconfig prerequisite (A: stop
TX before remoteproc resets WCSS; B: don't disable the ext IRQs twice
after a failed recovery). AP and mesh vifs on both radios, real
clients and mesh peers.

With your series + A:
- 10 simulated asserts (debugfs) while survey and station dumps are
  polled and downlink traffic runs: all recovered, no WARN/Oops.
- 10 real firmware asserts under load (5 AP vif, 5 mesh vif),
  triggered by an out-of-tree patch that sends WMI_PEER_FLUSH_TIDS_CMDID
  with all 32 TID bits set: all recovered, no WARN/Oops.
- Failed recovery (forced via a test-only knob) followed by another
  crash: the radios come back, but TX completions stop, TX fails with
  -12 and the firmware asserts again repeatedly, 2/2. That is what B
  fixes.

With your series + A + B:
- Failed recovery followed by another crash: recovered 3/3, which
  your 2/5 and 3/5 make possible in the first place.

One case is still open. It is not a regression of your series: before
3/5 the same failure freed the srng memory and crashed in the same
ath11k_ce_cleanup_pipes() path. If the firmware crashes again while
the restart work is bringing it up, the QMI firmware start can fail:

  remoteproc remoteproc0: remote processor cd00000.q6v5_wcss is now up
  remoteproc remoteproc0: crash detected in cd00000.q6v5_wcss: type fatal error
  ath11k c000000.wifi: fw_version 0x290b84a5 ...
  remoteproc remoteproc0: stopped remote processor cd00000.q6v5_wcss
  ath11k c000000.wifi: failed to send firmware start: -104
  ath11k c000000.wifi: failed to start firmware: -104
  ath11k c000000.wifi: failed to reconfigure driver on crash recovery

That recovery fails after ath11k_hal_srng_clear() but before
ath11k_ce_init_pipes(). The next recovery then tears down the CE pipes
on the cleared rings and hits a NULL pointer with BHs disabled (under
ce_lock):

  Unable to handle kernel read from unreadable memory at virtual address 0000000000000000
  pc : ath11k_hal_srng_access_begin+0xc/0x60 [ath11k]
  Kernel panic - not syncing: Oops: Fatal exception in interrupt

Decoded call chain (6.18 backport, the raw trace has wrong symbols
for the stripped module):

  ath11k_hal_srng_access_begin       hal.c:834
  ath11k_ce_poll_send_completed      ce.c:706
  ath11k_ce_cleanup_pipes            ce.c:884
  ath11k_core_reconfigure_on_crash   core.c:2403
  ath11k_core_restart                core.c:2574

Reproduced 2/2 by triggering a remoteproc crash 10-20 ms after "is now
up" (measured 11 and 21 ms), with your series and my two patches
applied. So 2/5 and 3/5 cover a recovery that fails once the rings are
set up again, but not a failure inside ath11k_core_start_firmware().

I haven't tried a fix yet. Since the CE-layer guard from v1 was
dropped, one option at the core level would be to remember whether the
CE pipes and DP were set up again since the last teardown, and to skip
that teardown in ath11k_core_reconfigure_on_crash() otherwise. I can
try that if you don't plan to cover it in a v3.

For the series:

Tested-by: Florian Maurer <fmaurer@xxxxxxxxxxx> # IPQ8074 hw2.0 AHB WLAN.HK.2.9.0.1-02146 (backport to 6.18)

[1] https://lore.kernel.org/linux-wireless/cover.1793531790.git.fmaurer@xxxxxxxxxxx/

Regards
Florian