Re: [PATCH v2 1/2] arm64: csum: Add fused copy and Internet checksum
From: Demian Shulhan
Date: Sun Nov 15 2026 - 12:25:42 EST
Hi David,
> Is that right?
> I thought ldp/stp required aligned addresses.
> The code below is using get_unaligned().
> I think the timings below are for ldp/stp?
Unlike 32-bit ARM (ldrd/strd), AArch64 GPR ldp/stp support unaligned
addresses in hardware on Normal memory. Both GCC 15 and Clang 21 merge
the adjacent get_unaligned()/put_unaligned() pairs into ldp/stp, and
the timings are from that code. Unaligned (src+1, dst+3) is only ~5%
slower due to cache-line splits (~28.8-29.1 GB/s vs ~30.3-31.1 GB/s).
> That is topping out at 10 bytes/clock.
> Each loop iteration copies 32 bytes - so must take 3 clocks.
> Which means it must be doing 16 byte read/write in a single clock.
> So rather begs the question of why it is limited by the loop control
> and checksum instructions rather than the memory accesses.
> The checksum code needs 4 ALU instructions (for 16 bytes), if the CPU
> can execute more than that then there is half a chance of getting the
> loop down to 2 clocks.
For 32 bytes, the loop is 16 instructions (4x ldp/stp, 8x adds/cinc,
2x add, cmp, b.ne). On Neoverse-V2 (6 ALU pipes, 8-wide dispatch) 12
ALU/branch ops plus 4 paired memory ops put the floor right around ~3
clocks/iteration (~5.2 IPC).
> Just moving one (or both) the pointer updates into the middle may help.
> Otherwise you try using negative offsets from the end of the buffer
> and 'reg + reg' addressing (will need more pointers since I don't think
> arm has 'reg + reg + offset' addressing.
Moving the pointer updates into the middle does help, it shaves up to
~5% off across sizes.
For 'reg + reg', ldp/stp only support immediate offsets, so indexed
addressing would force single-register ldr/str and double the memory
instructions.
> > + if (unlikely(len <= 0))
> > + return 0;
>
> Hmmm...
> Does zero actually happen often enough for that to be a gain?
> Even with the unlikely() it is likely to slow down the normal path.
> And that assumes that the branch isn't mispredicted and that gcc
> doesn't ignore the unlikely().
len == 0 would fall through fine, but since len is a signed int in the
csum_partial_copy_nocheck() prototype, this guards against negative len
wrapping (unsigned int)len >> 5 (matching do_csum() right above it).
Both compilers place the return 0 block out-of-line at the end of the
function.
> > + sum2 += v0;
> > + sum3 += v1;
>
> Is using sum0/1 here actually slower?
> My 'gut feel' is that it won't be - and saves the adds below.
For 64-128B your gut feel is right - 2 accumulators is 2-10% faster from
saving the post-loop fold, and on GCC 15 it matches 4 accumulators on
larger buffers too (GCC folds v0 + v2 first, keeping a 1-cycle chain in
18 insns). The catch is Clang 21: with 2 accumulators it chains both
adds/cinc updates onto the same accumulator and drops ~12% for >= 512B
(27.0 vs 30.5-31.2 GB/s).
> It is also worth checking that gcc doesn't zero extend v0 into a
> register and then do a full 128bit add.
> I'm sure I looked at the code using only 64bit variables for:
> sum0 += v0;
> carry0 += sum0 < v0;
> both gcc and clang used 'add 0 with carry' for the second line.
Checked - neither compiler zero-extends v0, both emit adds + cinc
directly on the 64-bit registers.
> > + s += 32;
> > + d += 32;
> > + } while (--n);
>
> It is a shame to have to update 3 variables here.
> They may be taking the 3rd clock.
Switching --n to an end pointer (s != end) and moving s += 32 / d += 32
between the loads and stores drops the 3rd update on Clang (80 -> 76
insns total) and cleans up GCC's loop setup (88 -> 81 insns), gaining
+0.4% to +4.9% across sizes.
Here is the resulting 16-instruction loop with an end pointer and 4
accumulators:
GCC 15.2: Clang 21.1:
ldp x10, x9, [x0] ldp x17, x18, [x0]
add x11, x11, #0x20 ldp x3, x4, [x0, #0x10]
ldp x8, x7, [x0, #16] add x0, x0, #0x20
add x0, x0, #0x20 adds x15, x15, x17
stp x10, x9, [x11, #-32] stp x17, x18, [x1]
adds x3, x10, x3 cinc x16, x16, hs
stp x8, x7, [x11, #-16] adds x13, x13, x18
cinc x12, x12, cs stp x3, x4, [x1, #0x10]
adds x4, x9, x4 cinc x14, x14, hs
cinc x13, x13, cs adds x8, x8, x3
adds x6, x8, x6 add x1, x1, #0x20
cinc x16, x16, cs cinc x9, x9, hs
adds x5, x7, x5 adds x10, x10, x4
cinc x15, x15, cs cinc x11, x11, hs
cmp x14, x0 cmp x0, x12
b.ne .Loop b.ne .Loop
I'll switch to the end-pointer loop with interleaved pointer updates in
v3.
Thanks!
Demian
сб, 14 лист. 2026 р. о 16:26 David Laight <david.laight.linux@xxxxxxxxx> пише:
>
> On Sat, 14 Nov 2026 03:57:00 +0000
> Demian Shulhan <demyansh@xxxxxxxxx> wrote:
>
> > arm64 currently falls back to the generic csum_partial_copy_nocheck() in
> > lib/checksum.c, which performs memcpy() followed by a second pass in
> > csum_partial(). This double pass is hit on software GSO paths
> > (skb_segment() without SG or TX checksum offload, e.g. TCP over
> > WireGuard) and UDP/ICMP replies.
> >
> > Implement a single-pass csum_partial_copy_nocheck() in
> > arch/arm64/lib/csum.c using four independent __uint128_t accumulators
> > over a 32-byte stride, followed by 16/8/4/2/1-byte tail handling. Each
> > word is read once into a GPR before storing and summing, emitting paired
> > ldp/stp and adds/cinc carry propagation with no stack frame.
>
> Is that right?
> I thought ldp/stp required aligned addresses.
> The code below is using get_unaligned().
> I think the timings below are for ldp/stp?
>
> >
> > Throughput on ARM Neoverse-V2 (Google Axion, 3.0 GHz, GCC 15.2 -O2,
> > warm L1 cache):
> >
> > len memcpy + do_csum csum_partial_copy_nocheck speedup
> > ---- ---------------- ------------------------- -------
> > 64 10245 MB/s 15902 MB/s +55.2%
> > 256 18851 MB/s 26026 MB/s +38.1%
> > 512 22465 MB/s 28421 MB/s +26.5%
> > 1024 24330 MB/s 29900 MB/s +22.9%
> > 1368 24192 MB/s 30221 MB/s +24.9%
> > 1448 24188 MB/s 30279 MB/s +25.2%
> > 1500 24165 MB/s 30096 MB/s +24.5%
> > 4096 25299 MB/s 29946 MB/s +18.4%
>
> That is topping out at 10 bytes/clock.
> Each loop iteration copies 32 bytes - so must take 3 clocks.
> Which means it must be doing 16 byte read/write in a single clock.
> So rather begs the question of why it is limited by the loop control
> and checksum instructions rather than the memory accesses.
> The checksum code needs 4 ALU instructions (for 16 bytes), if the CPU
> can execute more than that then there is half a chance of getting the
> loop down to 2 clocks.
> Just moving one (or both) the pointer updates into the middle may help.
> Otherwise you try using negative offsets from the end of the buffer
> and 'reg + reg' addressing (will need more pointers since I don't think
> arm has 'reg + reg + offset' addressing.
>
> >
> > Suggested-by: David Laight <david.laight.linux@xxxxxxxxx>
> > Signed-off-by: Demian Shulhan <demyansh@xxxxxxxxx>
> > ---
> > arch/arm64/include/asm/checksum.h | 3 +
> > arch/arm64/lib/csum.c | 100 ++++++++++++++++++++++++++++++
> > 2 files changed, 103 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/checksum.h b/arch/arm64/include/asm/checksum.h
> > index dc52b733675d..9f766db01152 100644
> > --- a/arch/arm64/include/asm/checksum.h
> > +++ b/arch/arm64/include/asm/checksum.h
> > @@ -44,6 +44,9 @@ static inline __sum16 ip_fast_csum(const void *iph, unsigned int ihl)
> > extern unsigned int do_csum(const unsigned char *buff, int len);
> > #define do_csum do_csum
> >
> > +#define _HAVE_ARCH_CSUM_AND_COPY
> > +__wsum csum_partial_copy_nocheck(const void *src, void *dst, int len);
> > +
> > #include <asm-generic/checksum.h>
> >
> > #endif /* __ASM_CHECKSUM_H */
> > diff --git a/arch/arm64/lib/csum.c b/arch/arm64/lib/csum.c
> > index 75db718e5b6a..a9617cde28c1 100644
> > --- a/arch/arm64/lib/csum.c
> > +++ b/arch/arm64/lib/csum.c
> > @@ -2,8 +2,10 @@
> > // Copyright (C) 2019-2020 Arm Ltd.
> >
> > #include <linux/compiler.h>
> > +#include <linux/export.h>
> > #include <linux/kasan-checks.h>
> > #include <linux/kernel.h>
> > +#include <linux/unaligned.h>
> >
> > #include <net/checksum.h>
> >
> > @@ -116,6 +118,104 @@ unsigned int __no_sanitize_address do_csum(const unsigned char *buff, int len)
> > return sum >> 16;
> > }
> >
> > +static __always_inline __wsum csum_fold128(__uint128_t sum)
> > +{
> > + u64 s;
> > +
> > + sum += (sum >> 64) | (sum << 64);
> > + s = sum >> 64;
> > + s += (s >> 32) | (s << 32);
> > + return (__force __wsum)(u32)(s >> 32);
> > +}
> > +
> > +/*
> > + * Single-pass copy and ones' complement sum. Each word is read once into a
> > + * GPR before storing and summing, so the returned sum always matches @dst
> > + * even if @src is concurrently modified.
> > + */
> > +__wsum csum_partial_copy_nocheck(const void *src, void *dst, int len)
> > +{
> > + const u8 *s = src;
> > + u8 *d = dst;
> > + __uint128_t sum0 = 0, sum1 = 0, sum2 = 0, sum3 = 0;
> > + unsigned int n;
> > +
> > + if (unlikely(len <= 0))
> > + return 0;
>
> Hmmm...
> Does zero actually happen often enough for that to be a gain?
> Even with the unlikely() it is likely to slow down the normal path.
> And that assumes that the branch isn't mispredicted and that gcc
> doesn't ignore the unlikely().
>
> > +
> > + n = (unsigned int)len >> 5;
> > + if (n) {
> > + do {
> > + u64 v0, v1;
> > +
> > + v0 = get_unaligned((const u64 *)s);
> > + v1 = get_unaligned((const u64 *)(s + 8));
> > + put_unaligned(v0, (u64 *)d);
> > + put_unaligned(v1, (u64 *)(d + 8));
> > + sum0 += v0;
> > + sum1 += v1;
> > +
> > + v0 = get_unaligned((const u64 *)(s + 16));
> > + v1 = get_unaligned((const u64 *)(s + 24));
> > + put_unaligned(v0, (u64 *)(d + 16));
> > + put_unaligned(v1, (u64 *)(d + 24));
> > + sum2 += v0;
> > + sum3 += v1;
>
> Is using sum0/1 here actually slower?
> My 'gut feel' is that it won't be - and saves the adds below.
>
> It is also worth checking that gcc doesn't zero extend v0 into a
> register and then do a full 128bit add.
> I'm sure I looked at the code using only 64bit variables for:
> sum0 += v0;
> carry0 += sum0 < v0;
> both gcc and clang used 'add 0 with carry' for the second line.
>
> > +
> > + s += 32;
> > + d += 32;
> > + } while (--n);
>
> It is a shame to have to update 3 variables here.
> They may be taking the 3rd clock.
>
> David
>
>
> > +
> > + sum0 += sum2;
> > + sum1 += sum3;
> > + }
> > +
> > + if (len & 16) {
> > + u64 v0 = get_unaligned((const u64 *)s);
> > + u64 v1 = get_unaligned((const u64 *)(s + 8));
> > +
> > + put_unaligned(v0, (u64 *)d);
> > + put_unaligned(v1, (u64 *)(d + 8));
> > + sum0 += v0;
> > + sum1 += v1;
> > + s += 16;
> > + d += 16;
> > + }
> > + if (len & 8) {
> > + u64 v = get_unaligned((const u64 *)s);
> > +
> > + put_unaligned(v, (u64 *)d);
> > + sum0 += v;
> > + s += 8;
> > + d += 8;
> > + }
> > + if (len & 4) {
> > + u32 v = get_unaligned((const u32 *)s);
> > +
> > + put_unaligned(v, (u32 *)d);
> > + sum1 += v;
> > + s += 4;
> > + d += 4;
> > + }
> > + if (len & 2) {
> > + u16 v = get_unaligned((const u16 *)s);
> > +
> > + put_unaligned(v, (u16 *)d);
> > + sum0 += v;
> > + s += 2;
> > + d += 2;
> > + }
> > + if (len & 1) {
> > + u8 v = *s;
> > +
> > + *d = v;
> > + sum1 += v;
> > + }
> > +
> > + return csum_fold128(sum0 + sum1);
> > +}
> > +EXPORT_SYMBOL(csum_partial_copy_nocheck);
> > +
> > __sum16 csum_ipv6_magic(const struct in6_addr *saddr,
> > const struct in6_addr *daddr,
> > __u32 len, __u8 proto, __wsum csum)
>