[PATCH] mm/madvise: add cond_resched() in THP branch of madvise_cold_or_pageout_pte_range()

[albin_yang]

From: Wei Yang <albinwyang@xxxxxxxxxxx>

During stress testing with stress-ng on a 256-CPU system, a softlockup
was observed when madvise(MADV_COLD) was called on a 31GB memory region
backed by transparent hugepages. The region contains 15935 THP PMD
entries, and the softlockup was triggered after 14485 of them had been
processed.

The THP branch of madvise_cold_or_pageout_pte_range() has no scheduling
point. walk_pmd_range() invokes it once per 2MB PMD entry with no
cond_resched() anywhere in the loop. Each iteration calls
pmdp_invalidate() which issues a synchronous TLB shootdown (dsb ish)
waiting for all CPUs to acknowledge. This operation can be
time-consuming under heavy system load, causing the accumulated time
across all PMD entries to exceed the softlockup threshold (20s) and
panic when softlockup_panic=1. On a CONFIG_PREEMPT_NONE=y kernel, there
is no other opportunity to reschedule during this entire traversal.

The regular PTE path already has an unconditional cond_resched() at the
function tail (introduced by commit 9c276cc65a58 ("mm: introduce
MADV_COLD") and commit 1a4e58cce84e ("mm: introduce MADV_PAGEOUT") when
MADV_COLD and MADV_PAGEOUT were first introduced) and a conditional one
inside the loop (commit b2f557a21bc8 ("mm/madvise: add cond_resched()
in madvise_cold_or_pageout_pte_range()")). The THP branch never
received equivalent treatment.

Fix by adding cond_resched() at the huge_unlock exit path after
spin_unlock(ptl) and reclaim_pages(), which is the single exit point
for all goto paths in the THP branch.

Crash log:
  watchdog: BUG: soft lockup - CPU#63 stuck for 23s! [stress-ng-vm:1665914]
  ...
  Call trace:
  pmdp_invalidate+0x48/0x68
  madvise_cold_or_pageout_pte_range+0x84c/0xb00
  walk_pmd_range.isra.0+0x1ac/0x3b0
  walk_pud_range+0x120/0x190
  walk_pgd_range+0x170/0x1b8
  __walk_page_range+0x184/0x198
  walk_page_range_vma+0x94/0x128
  madvise_cold+0xd8/0x198
  madvise_vma_behavior+0x234/0x638
  madvise_walk_vmas.constprop.0+0xdc/0x210
  madvise_do_behavior+0xa8/0x328
  do_madvise+0xd8/0x140
  __arm64_sys_madvise+0x2c/0xe0

Signed-off-by: Wei Yang <albinwyang@xxxxxxxxxxx>
---
 mm/madvise.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/madvise.c b/mm/madvise.c
index eeee82cf2b3f..0755e6d99757 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -450,6 +450,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 		spin_unlock(ptl);
 		if (pageout)
 			reclaim_pages(&folio_list);
+		cond_resched();
 		return 0;
 	}
 
-- 
2.43.5