[PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies") stopped selecting REP MOVSB from ERMS alone because it
is not suitable for short copies. Consequently, ordinary out-of-line
__memcpy() calls on CPUs with ERMS but without FSRM such as Cooper Lakes
and Skylakes use memcpy_orig() for every length. Its main loop copies
32 bytes per iteration and can be slower than REP MOVSB for large copies,
as measured below on Cooper Lake.

This matters for btrfs zstd decompression. The btrfs-tree commit
e27a6e4823f3 ("btrfs: zstd: avoid a copy in zstd_decompress_bio()")
removes btrfs' copy from a scratch buffer into the page cache by letting
zstd write there directly. It does not remove the copies inside
streaming zstd. ZSTD_decompressStream() still uses memcpy() to flush
decompressed data from its internal output buffer to the supplied page
(zdss_flush), and to stage compressed input when the next block crosses
an input folio supplied by btrfs (zdss_load). In Meta's fleet profile
of x86 kernels, these two btrfs/zstd call sites accounted for about 34%
of the sampled profile weight attributed to memcpy_orig().

On CPUs without FSRM, select REP MOVSB for copies of 2048 bytes and above
when ERMS is present. Keep memcpy_orig() below 2048 bytes. The later FSRM
alternative continues to select REP MOVSB for all sizes, including when
ERMS is not exposed. Copies below 2048 bytes were showing regression
with REP MOVSB as seen below (medians from five runs):

  bytes          orig    REP MOVSB   increase
     32       10.9 ns      23.1 ns     112.3%
     64       13.9 ns      23.1 ns      66.3%
    128       17.5 ns      28.9 ns      65.2%
    256       26.0 ns      31.7 ns      22.1%

Dispatch ERMS through a separate helper and put the destination save in
the FSRM replacement. This keeps the entry alternative at five bytes and
preserves memcpy_orig()'s instructions and placement. FSRM executes the
destination move and one two-byte NOP before its direct REP MOVSB path.

Userspace replicas of the original and patched entry paths were run on
a turbo-disabled Xeon Platinum 8321HC (Cooper Lake, ERMS without FSRM),
at 1.4 GHz. Results below are median times per call from five process
runs of aligned 4 KiB copies at entry offset zero. The replicas use
plain RET; these are assembly microbenchmark results.

Here "warmed" means repeatedly using both buffers. The other rows flush
every cache line intersecting the source or destination before timing.

                                    orig    patched  reduction
  warmed                        371.6 ns   123.4 ns      66.8%
  source flushed                706.5 ns   400.8 ns      43.3%
  destination flushed           457.7 ns   350.0 ns      23.5%

Signed-off-by: Usama Arif <usama.arif@xxxxxxxxx>
---
 arch/x86/lib/memcpy_64.S | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..6b3e203b6087c 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -31,9 +31,10 @@
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
+	/* The FSRM destination move shares the space used by the ERMS jump. */
+	ALTERNATIVE_2 "jmp memcpy_orig", "jmp memcpy_erms", X86_FEATURE_ERMS, \
+		      "movq %rdi, %rax", X86_FEATURE_FSRM
 
-	movq %rdi, %rax
 	movq %rdx, %rcx
 	rep movsb
 	RET
@@ -171,3 +172,11 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	RET
 SYM_FUNC_END(memcpy_orig)
 
+SYM_FUNC_START_LOCAL(memcpy_erms)
+	cmpq $2048, %rdx
+	jb memcpy_orig
+	movq %rdi, %rax
+	movq %rdx, %rcx
+	rep movsb
+	RET
+SYM_FUNC_END(memcpy_erms)
-- 
2.53.0-Meta


> 
> Why 512? memmove() uses REP MOVSB from 32 bytes on ERMS. Can we do the
> same here? Then we don't need a new branch at all.
> 
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -50,6 +50,8 @@
>  	cmpq $0x20, %rdx
>  	jb .Lhandle_tail
>  
> +	ALTERNATIVE "", __stringify(movq %rdx, %rcx; rep movsb; RET), X86_FEATURE_ERMS
> +
>  	/*
>  	 * We check whether memory false dependence could occur,
>  	 * then jump to corresponding copy mode.
> -- 
>   Kiryl Shutsemau / Kirill A. Shutemov
>