[PATCH v4 0/2] arm64: speed up __arch_copy_from_user() for large copies

From: Qi Xi

Date: Mon Nov 16 2026 - 02:31:59 EST


From: Qi Xi <xiqi2@xxxxxxxxxx>

Based on Ben Niu's "Faster Arm64 __arch_copy_from_user and
__arch_copy_to_user" work [1], this series continues the optimization
for __arch_copy_from_user() only, following Catalin's review of v3 [2].

For copies of 4096 bytes or more, __arch_copy_from_user() uses
privileged LDP instead of LDTR and toggles PSTATE.PAN for the duration
of the copy, together with an offset-addressed 64-byte loop added to
copy_template.S. Smaller copies keep the original code.

copy_to_user() is deliberately left alone. Clearing PAN and using
privileged STP there puts the PAN update in the middle of the user
store stream, which breaks hardware write streaming; on Kunpeng 920
the resulting regression outweighs the gain from paired stores.

Performance improvements measured on Kunpeng 920:

The ku_copy microbenchmark [3] (a kernel module that measures
copy_from_user throughput across various sizes by copying 1GB of
data in each test):
copy_from_user throughput change (positive = improvement):
8B-2048B: ~0% (not covered by this optimization, no regression)
4KB: +2.0% 8KB: +19.1% 16KB: +29.8% 32KB: +36.5%
64KB: +38.0% 128KB: +57.9% 256KB: +71.2%

Real-world workloads:
- iperf3 over virtio-net/tap in a two-level PVM stack
(L1 send to KVM L2 guest):
Throughput improved from 38.5 to 50.1 Gbps (+30%).
copy_from_user hotspot in the QEMU main-loop thread reduced
from 68% to 55%.
- meshproxy:
256KB request latency reduced from 0.91 ms to 0.78 ms (-14%)
at 16 connections / 16k QPS.

Changes since v3:
- No global PAN disabling. PAN is toggled only around the
new fast path, so LDTR/STTR remain the default for every
other copy and keep protecting against user-controlled kernel
accesses, as Catalin pointed out.
- Fault semantics fixed: every load in the 64-byte loop has its own
exception table entry, and the fixup advances dst by the number of
16-byte blocks that were actually stored. A fault in the middle of
the loop now returns the number of bytes really not copied instead.
- A 4096-byte threshold is added; from there on the new path is
used, since toggling PAN also has a cost.
- copy_to_user() is deliberately left alone.

[1] https://lore.kernel.org/all/20251018052237.1368504-2-benniu@xxxxxxxx/
[2] https://lore.kernel.org/all/20260316123100.82932-1-xiqi2@xxxxxxxxxx/
[3] https://github.com/mcfi/benchmark/tree/main/ku_copy

Qi Xi (2):
arm64: copy_template: add an offset-addressed 64-byte loop
arm64: copy_from_user: use privileged LDP/STP for large copies

arch/arm64/lib/Makefile | 2 +-
arch/arm64/lib/copy_from_user.S | 8 ++
arch/arm64/lib/copy_from_user_opt.S | 114 ++++++++++++++++++++++++++++
arch/arm64/lib/copy_template.S | 35 +++++++++
4 files changed, 158 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/lib/copy_from_user_opt.S

--
2.53.0