[PATCH v3 0/6] mm: access remote process memory under the per-VMA lock

[Rik van Riel]

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines with large multi-threaded applications, the mmap_lock
is often contended due to mixed accesses from readers and writers, like
mmap and munmap. When a lock holder gets stuck, system monitoring software
can get stuck behind that, resulting in a failure to log that the system
is in trouble.

Take the per-VMA lock in __access_remote_vm() when the access falls
entirely within a single VMA. Fall back to the mmap lock when the access
crosses a VMA boundary, or when the page cannot be reached under the
per-VMA lock: a dropped fault, a userfaultfd VMA, a hard error, or memory
with no struct page that has to go through vma->vm_ops->access().

The bulk of the work is a new gup helper. __access_remote_vm() needs a
single page from a VMA it has already looked up and locked, faulting it in
when necessary, under either lock.

get_user_pages_remote() does not fit: it hard codes the mmap lock and
re-derives the VMA. get_user_page_vma() walks the page tables, faults a
missing page in, and returns it with a reference and the caller's lock
still held.

The per-VMA path also closes a pre-existing gap. A COWed page in a
VM_IO/VM_PFNMAP VMA has a struct page, but the old code routed it to
->access(), where generic_access_phys() ioremaps the PFN and ioremap of RAM
is rejected, so the read came up short.

get_user_page_vma() now returns that page normally. Raw PFNs with no struct
page still reach ->access() under the mmap lock, as before.

The series is arranged as:

  1-2: untag the remote address in the VMA lookup without the mmap lock,
       on x86 and riscv.
  3:   rename get_user_page_vma_remote() to get_user_page_lookup_vma().
  4:   add get_user_page_vma().
  5:   switch __access_remote_vm() to the per-VMA lock.
  6:   add selftest coverage.

Changes since v2 [1]. The per-VMA fast path is reworked to build on the GUP
lookup+fault path instead of a custom walker, per David Hildenbrand's
and Lorenzo Stoakes' review, and now faults pages in rather than only
reading resident ones.

  - Drop the folio_walk_start(FW_VMA_LOCKED) walk. Add get_user_page_vma()
    in mm/gup.c (patch 4): a trimmed __get_user_pages() that walks with
    follow_page_mask(), faults in with faultin_page(), and returns the page
    with the caller's lock held. Safe under the per-VMA lock because page
    tables are RCU-freed, so interrupts need not be disabled. (David)
  - Fault pages in on the fast path. v2 fell back to the mmap lock for any
    not-present page; now it falls back only on -EAGAIN. (David)
  - Turn the v2 READ_ONCE/WRITE_ONCE untag_mask change into an
    untagged_addr_remote_unlocked() helper (patch 1), and add the riscv
    pointer-masking equivalent (patch 2), which v2 did not cover.
  - Read COWed pages in VM_IO/VM_PFNMAP VMAs; raw PFNs still fall back to
    ->access() under the mmap lock.
  - Add the get_user_page_lookup_vma() rename (patch 3) and selftest
    coverage for the struct-page and raw-PFN paths (patch 6).

  [1] https://lore.kernel.org/all/20260625015053.2445008-1-riel@xxxxxxxxxxx/

Rik van Riel (6):
      x86/mm: add untagged_addr_remote_unlocked()
      riscv/mm: add untagged_addr_remote_unlocked()
      mm: rename get_user_page_vma_remote() to get_user_page_lookup_vma()
      mm/gup: add get_user_page_vma() to fault in a page under a held lock
      mm: use per-VMA lock in __access_remote_vm() for single-VMA accesses
      selftests/mm: cover /proc/pid/mem access to VM_PFNMAP memory

 arch/arm64/kernel/mte.c              |   2 +-
 arch/riscv/include/asm/mmu_context.h |   4 +-
 arch/riscv/include/asm/uaccess.h     |  10 +-
 arch/riscv/kernel/process.c          |  12 +-
 arch/x86/include/asm/mmu_context.h   |   6 +-
 arch/x86/include/asm/uaccess_64.h    |  14 ++-
 arch/x86/kernel/process_64.c         |   4 +-
 arch/x86/kernel/uprobes.c            |   2 +-
 include/linux/mm.h                   |   2 +-
 include/linux/uaccess.h              |   7 ++
 mm/gup.c                             | 156 ++++++++++++++++++++++----
 mm/internal.h                        |   7 +-
 mm/memory.c                          | 171 ++++++++++++++++++++-------
 mm/rmap.c                            |   2 +-
 tools/testing/selftests/mm/pfnmap.c  |  66 +++++++++++
 15 files changed, 375 insertions(+), 90 deletions(-)

base-commit: 0f26556c5eeea62cc934fa8938b148aa5844a6b6
--
2.47.0