[PATCH 0/2] SUNRPC: Recycle sent Reply pages instead of freeing them

[Ameer Hamza]

An NFS READ Reply built in pages the nfsd thread allocated has
those pages freed from softirq, when the peer's ACK arrives. Since
commit 574907741599 ("mm/page_alloc: leave IRQs enabled for per-cpu
page allocations") such frees collide with alloc_pages_bulk()'s
IRQs-on pcp window and fall back to free_one_page() under
zone->lock; on a single memory node every thread contends for the
same zone->lock and the collisions compound. Replies spliced from
page-cache folios are immune, since the page cache still holds a
reference and the ACK-time put never reaches the allocator. The
rest are not. On gfs2, kernfs, cifs direct I/O and DAX files
splicing runs copy_splice_read(), which allocates a fresh page for
every page of payload, and nfsd allocates them too when it does not
splice at all: under nfsd_disable_splice_read, and for sec=krb5i
and sec=krb5p.

Patch 1 removes a redundant allocation: on those files nfsd builds
the Reply out of pages copy_splice_read() allocated for that one
Reply and discards the pages it was already holding, so it
allocates twice for every page of payload. Reading through
nfsd_iter_read() does the same single copy straight into the pages
nfsd already owns. Patch 2 removes the free: nfsd drops its own
reference as soon as the Reply is sent, leaving the network with
the last one. Keeping that reference instead and reusing the page
once folio_ref_count() reads 1 keeps it out of the allocator. A
thread holds at most 4 MiB of pages, trimmed as its own demand
falls and freed when it exits.

How much this is worth depends on contention: on the server below
the ACK-time frees reach 43% of cycles, while on an uncontended
machine the same change removes the allocations and leaves total
CPU where it was.

The measurements below use ext4 with DAX and tmpfs so they
reproduce on a stock tree with a pmem device; ZFS is affected the
same way, its ->splice_read being copy_splice_read() as well, but
nothing here depends on it. Single-node server, 64 threads, one
nfsd thread per CPU, eight clients, 1 MiB sequential cached reads
over NFSv4.0, 60s a run, clients in network namespaces on the same
box, so vmstat counts their allocations too.

                              cycles in       pages allocated
                            free_one_page    per payload page
  ext4 dax  NFSv4            43%  ->   0%        2.97 -> 0.05
  ext4 dax  NFSv4, knob      32%  ->   0%        1.99 -> 0.05
  tmpfs     NFSv4, knob      33%  ->   0%        1.98 -> 0.05

The knob rows have nfsd_disable_splice_read set, so they reach the
same path without patch 1 and measure patch 2 alone. The first
column is free_one_page()'s share of all CPU cycles on the box, the
second is vmstat page allocations divided by the READ payload
delivered, Call buffers included.

The table is from the series backported to 6.18.38, where the
release path is adapted to that tree's combined rq_pages array.
The posted code was run only in a 20-CPU VM, not on that server.
free_one_page() falls from 24628 calls per million pages served
to 5990 and softirq time per page from 0.836 to 0.528 us, while
total CPU per page does not move: svc_reuse_scan()'s 2.5% of
cycles is paid back by the allocator work it removes. The
scan's share falls to 0.7% when late ACKs leave little to
reuse, since it stops after a pass in which no page is ready.

Both read paths pass over NFSv3 and NFSv4.2 under KASAN, lockdep
and DEBUG_VM, with xfstests -g quick and pynfs 4.0/4.1 matching
the unpatched baseline, cthon04 passing, and byte-identical
checksums over NFS/RDMA and NFSv3/UDP. All of that runs on a fast
link, where a build with the folio_ref_count() gate deliberately
removed also passes the checksum runs. Under a 40 ms ACK delay
that build corrupts data and the series stays byte-identical.

Ameer Hamza (2):
  NFSD: Use nfsd_iter_read() when ->splice_read is not zero-copy
  SUNRPC: Recycle sent Reply pages instead of freeing them

 fs/nfsd/nfs4xdr.c               |   4 +-
 fs/nfsd/vfs.c                   |   5 +-
 fs/nfsd/vfs.h                   |  29 ++++
 include/linux/sunrpc/svc.h      |  14 ++
 include/linux/sunrpc/svc_xprt.h |  14 ++
 include/trace/events/sunrpc.h   |  32 ++++-
 net/sunrpc/svc.c                | 235 +++++++++++++++++++++++++++++++-
 net/sunrpc/svc_xprt.c           |   7 +
 net/sunrpc/svcsock.c            |   2 +
 9 files changed, 335 insertions(+), 7 deletions(-)


base-commit: 76427d869120552a1a82e1f1488d9f8311827d84
-- 
2.53.0