[PATCH 0/5] mm/khugepaged: bound mmap_lock hold time when walking an mm's VMAs

[Lorenzo Stoakes (ARM)]

This series introduces a couple small changes to khugepaged to improve its
behaviour both in the face of mm->mmap_lock contention and to reduce that
lock contention when possible.

It also updates other places which release the mmap read lock to allow mmap
contended mmap write locks a chance to make progress (BPF iterators and
/proc/$pid/maps + friends).

Looking at the first change - if the mmap lock is contended, an attempted
mmap trylock in collapse_scan_mm_slot() will result in the scan being
aborted wherever it happens to be, and when next the mm is processed it
starts from the beginning again.

If an mm has a lot of ineligible VMAs and is undergoing significant
mmap_lock contention, collapsing the eligible VMAs there can take a very
long time.

Patch 2 of the series fixes this (patch 1 is a prerequisite) by keeping
track of where khugepaged was in the mm before aborting the scan.

This issue was noticed during some work in which an mm was instantiated
with a large number of small VMAs under significant mmap_lock contention,
during which collapse was observed to take 10.8s. With this patch applied
it took 3.6s to collapse the same mm.

Patch 3 introduces new rwsem primitives - down_read_nosteal() and
down_read_killable_nosteal() - to allow for read locks to be taken without
stealing them from a queued write lock.

Patch 4 adds mmap lock wrappers for this - mmap_read_lock_nosteal() and
mmap_read_lock_killable_nosteal() and updates callers which release the mmap
read lock on contention - BPF iterators + /proc/$pid/maps et al.

Finally, patch 5 uses this to have khugepaged release + reacquire the mmap
read lock in the face of a large number of ineligible VMAs.

This was also observed as part of the previously mentioned work in the form
of mmap lock stalls, which patch 5 resolves.

Importantly, none of the changes introduces any regressions for khugepaged
anywhere. It is all upside, and implemented conservatively.

Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
---
Lorenzo Stoakes (ARM) (5):
      Revert "mm/khugepaged: remove definition of struct khugepaged_mm_slot"
      mm/khugepaged: scan mm's more efficiently under mmap_lock contention
      locking/rwsem: add down_read[_killable]_nosteal()
      mm: add and use mmap_read_lock[_killable]_nosteal() on contention
      mm/khugepaged: drop/reacquire mmap read lock if skipped enough VMAs

 include/linux/mmap_lock.h |  44 +++++++++++++++++++
 include/linux/rwsem.h     |   2 +
 kernel/bpf/task_iter.c    |   2 +-
 kernel/locking/rwsem.c    |  71 ++++++++++++++++++++++++------
 mm/khugepaged.c           | 108 ++++++++++++++++++++++++++++++----------------
 mm/mmap_lock.c            |   2 +-
 6 files changed, 178 insertions(+), 51 deletions(-)
---
base-commit: 34f88a114888c9dba49e45f2b0c2a3e83c4c871c
change-id: 20261029-khugepaged-latency-fix-40495d66ea50

Cheers,
-- 
Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>