[PATCH 1/2] mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()

From: Seunghyun An

Date: Mon Nov 09 2026 - 19:10:36 EST


__vm_anon_pgoff_hi sits right after vm_refcnt in the vm_refcnt
cacheline. Since commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track
MAP_PRIVATE file-backed anon folios"), which made anon rmap use it,
__folio_set_anon() reads it on every new anonymous folio, i.e. once per
fault that maps a new anonymous page, between vma_start_read() and
vma_end_read().

When several threads fault on the same VMA, the other CPUs keep writing
vm_refcnt from vma_start_read()/vma_end_read(), so this read often
misses and pulls the line over. The extra load can force a coherence
transfer and makes the following vm_refcnt updates more expensive.

For anonymous VMAs the anonymous page offset is always equal to vm_pgoff
(linear_anon_page_index() already asserts this under CONFIG_DEBUG_VM;
it holds for MAP_PRIVATE /dev/zero too since commit e59c9a971a9f
("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous")),
so compute the index from vm_pgoff as before. That brings back the
pre-series behaviour for the hot path; MAP_PRIVATE file-backed VMAs still
read the split field. Keep the assertion, the other way round.

Moving __vm_anon_pgoff_hi out of the line does not work without growing
vm_area_struct. With e.g. CONFIG_ANON_VMA_NAME=y the struct is exactly
192 bytes, cachelines 0 and 1 have no hole, and the only 4-byte fields
that could fill the slot next to vm_refcnt are vm_lock_seq and the lo/hi
halves. vm_lock_seq is read before vm_refcnt is taken in vma_start_read(),
so it would just move the miss.

Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on
mainline after v7.4-rc1 (the base of this series). Threads share one
anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED;
faults/s, median of 3:

CPUs base patched
1 CCD, 7 threads 2.360M 2.369M +0.4%
7 CCDs, 1 thread each 1.738M 1.982M +14.0%
12 CCDs, 1 thread each 2.288M 2.548M +11.4%

On 7.3.1 (with the 7.3 variant of this change) the same runs gave +1.9%,
+16.0% and +29.4%. There, perf c2c shows the folio_add_new_anon_rmap()
load at offset 132 disappearing from the vm_refcnt line and the average
load latency of the vm_refcnt inc/dec on that line dropping by about
45%. With one VMA per thread, or with processes, this patch alone
changes throughput by -2.0% to -0.1% on that base (less than 1% on 7.3.1);
the next patch more than recovers that.

will-it-scale page_fault1 threads does not show the difference on this
machine: each thread mmap()s and munmap()s its own region, which merges
VMAs and is dominated by VMA write locking, and its results vary by up
to 10% between boots of the same kernel.

This relies on the /dev/zero commit above, which is not in v7.3, so this
is not tagged for stable: without it, MAP_PRIVATE /dev/zero VMAs are
anonymous but keep the file offset in vm_pgoff, and must keep using the
anon pgoff field.

The tools/testing/vma copy of the helper is updated to match.

Signed-off-by: Seunghyun An <san55@xxxxxxxx>
---
include/linux/pagemap.h | 19 ++++++++++++++-----
tools/testing/vma/include/dup.h | 13 ++++++++-----
2 files changed, 22 insertions(+), 10 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 73af18a373..300ee83fc1 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1122,13 +1122,22 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
const unsigned long address)
{
- const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
- if (vma_is_anonymous(vma))
- VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));

- return pgoff;
+ /*
+ * For anonymous VMAs the anonymous page offset always equals vm_pgoff,
+ * so use it. This keeps the anonymous fault path (__folio_set_anon())
+ * off __vm_anon_pgoff_hi, which shares a cacheline with vm_refcnt and
+ * is therefore contended by concurrent faults on the same VMA.
+ */
+ if (vma_is_anonymous(vma)) {
+ const pgoff_t pgoff = linear_page_index(vma, address);
+
+ VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+ return pgoff;
+ }
+
+ return __linear_anon_page_index(vma, address);
}

struct wait_page_key {
diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h
index dc4927d262..f3d69b0c74 100644
--- a/tools/testing/vma/include/dup.h
+++ b/tools/testing/vma/include/dup.h
@@ -1655,13 +1655,16 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
const unsigned long address)
{
- const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
- if (vma_is_anonymous(vma))
- VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));

- return pgoff;
+ if (vma_is_anonymous(vma)) {
+ const pgoff_t pgoff = linear_page_index(vma, address);
+
+ VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+ return pgoff;
+ }
+
+ return __linear_anon_page_index(vma, address);
}

extern const struct file_operations zero_fops;
--
2.53.0