[PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies") left ERMS CPUs without FSRM, including Cooper Lake and
Skylake, using scalar code for all out-of-line copies. Btrfs/zstd input
staging and output flushing accounted for about 34% of memcpy_orig()'s
sampled weight in Meta's x86 fleet profile.

Use REP MOVSB from 2048 bytes on ERMS CPUs without FSRM, keeping scalar
copies below that threshold. FSRM continues to use REP for all sizes.

Merge scalar code into __memcpy() and overlay its existing 32-byte check
with REP MOVSB and RET on FSRM, removing the non-FSRM entry jump. Copies
below 32 bytes bypass the ERMS gate. SHR11/JNZ fits one ten-byte NOP
without ERMS; the REP arm reloads the original count from RDX.

With ITS configured, align replacement storage before its start label so
the raw FSRM return retains its thunk before relocation. The alignment
padding is outside the copied range.

On Cooper Lake (boost disabled, capped at 1.401 GHz), plain-RET userspace
replicas with aligned warmed buffers and entry offset zero modulo 64
measured 4 KiB copies at 371.7 ns originally and 123.4 ns with this patch:
a 66.8% paired reduction across five process runs. A separate sweep using
REP from 32 bytes increased 32/64/128/256-byte times by
114.9%/54.4%/66.1%/23.2%, respectively.

Small-copy costs remain: a warmed aligned zero-byte copy at entry offset
48 modulo 64 took a paired 0.75 ns more (+7.9%) than the original, in all
five runs. These microbenchmarks do not establish a kernel workload gain
or the best threshold across CPUs.

Direct copy checks and memcpy KUnit passed with all four ERMS/FSRM
combinations and with ITS forced on; the copied return target was verified.

Signed-off-by: Usama Arif <usama.arif@xxxxxxxxx>
---
 arch/x86/lib/memcpy_64.S | 42 ++++++++++++++++++++++++++--------------
 1 file changed, 27 insertions(+), 15 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..407b226d4de2f 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -31,24 +31,27 @@
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
-
 	movq %rdi, %rax
 	movq %rdx, %rcx
-	rep movsb
-	RET
-SYM_FUNC_END(__memcpy)
-EXPORT_SYMBOL(__memcpy)
-
-SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
-SYM_PIC_ALIAS(memcpy)
-EXPORT_SYMBOL(memcpy)
 
-SYM_FUNC_START_LOCAL(memcpy_orig)
-	movq %rdi, %rax
+#ifdef CONFIG_MITIGATION_ITS
+	/*
+	 * Returns are patched before alternatives. Keep the replacement
+	 * return in the first half of a cache line so ITS retains its thunk
+	 * when the instruction is copied to its final address.
+	 */
+	.pushsection .altinstr_replacement, "ax"
+	.balign 64
+	.popsection
+#endif
+	ALTERNATIVE __stringify(cmpq $0x20, %rdx; jb .Lhandle_tail), \
+		    __stringify(rep movsb; RET), X86_FEATURE_FSRM
 
-	cmpq $0x20, %rdx
-	jb .Lhandle_tail
+	/*
+	 * Prepared RCX is scratch for checking length >= 2048.
+	 * The 10-byte alternative fits a NOP without ERMS.
+	 */
+	ALTERNATIVE "", __stringify(shrq $11, %rcx; jnz .Lerms), X86_FEATURE_ERMS
 
 	/*
 	 * We check whether memory false dependence could occur,
@@ -169,5 +172,14 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 
 .Lend:
 	RET
-SYM_FUNC_END(memcpy_orig)
 
+.Lerms:
+	movq %rdx, %rcx
+	rep movsb
+	RET
+SYM_FUNC_END(__memcpy)
+EXPORT_SYMBOL(__memcpy)
+
+SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
+SYM_PIC_ALIAS(memcpy)
+EXPORT_SYMBOL(memcpy)
-- 
2.53.0-Meta