Re: [PATCH 0/2] arm64: csum: Add fused copy and Internet checksum

[David Laight]

On Tue, 3 Nov 2026 16:47:29 +0100
Demian Shulhan <demyansh@xxxxxxxxx> wrote:

> > .L4:
> >         ldr     x1, [x0, x4]
> >         adds    x2, x2, x1
> >         cinc    x3, x3, cs      # adc $0,%rxx on x86
> >         adds    x4, x4, 8
> >         bne     .L4  
> Agreed, that is a nice loop for the sum-only case, one dependent op on
> the sum chain and no builtin needed.
> For the copy case I tried the same form on x86 (Zen 3, GCC 15, Clang
> 19-22, -O2) with a store added, against csum-copy_64.S, as posted it ran
> at 0.85x of the asm with GCC and 0.73-0.76x with Clang (which rewrites
> it into setb/adc), and ~0.8x unrolled 2x/4x - a bit slower than the
> posted 4-accumulator loop (0.88-0.95x).

I just added *(u64 *)(dest + count) = val to one of my loops.
gcc just replaces the 'add from memory' with mov/mov/add.

I'm now confused though, I seem to be getting different output from earlier.
With -Os (which I didn't think I was passing) clang is refusing to do the
'offset from end of buffer' loop.
With -O2 it bloats everything by unrolling 4 times and adding lots of
extra instructions.
(AFAICT the optimiser has a completely incorrect cpu model.)

If the compiler doesn't generate the expected code it is all pointless.
The xor parity code loop had to be in the asm block to stop the compiler
changing it to a much slower form.

For the checksum code I have managed to work out how many bytes/clock
the code was generating and getting it to (reasonably) match the expected
values.
But you have to use relatively short buffers to avoid D-cache effects.

David

> 
> пн, 2 лист. 2026 р. о 17:14 David Laight <david.laight.linux@xxxxxxxxx> пише:
> 
> 
> >
> > On Mon, 2 Nov 2026 16:14:06 +0100
> > Demian Shulhan <demyansh@xxxxxxxxx> wrote:
> >  
> > > > The scalar path seems like generic C code and not restricted to arm64,
> > > > curious how that performs against existing (e.g. x86) assembly
> > > > optimized implementation.  
> > > Depends. I think __builtin_addcll() is needed to get roughly the same
> > > performance as the asm version. In my tests GCC on arm64 turns it
> > > into adds + cset instead of adcs like on x86.
> > > But I will look at it, thanks!  
> >
> > The C version I posted earlier has the loop:
> >     buf += count;
> >     count = -count;
> >     sum = carry = 0;
> >     do {
> >         val = *(u64 *)(buf + count);
> >         sum += val;
> >         carry += sum < val;
> >     } while ((count += 8));
> >
> > for aarch64 you get:
> > .L4:
> >         ldr     x1, [x0, x4]
> >         adds    x2, x2, x1
> >         cinc    x3, x3, cs      # adc $0,%rxx on x86
> >         adds    x4, x4, 8
> >         bne     .L4
> >
> > I suspect that is close to optimal - especially for fast cpu with
> > out of order execution.
> >
> > https://godbolt.org/z/eso79K7Wq
> >
> > David
> >  
> > >
> > > пн, 2 лист. 2026 р. о 12:44 saeed bishara <saeed.bishara.os@xxxxxxxxx> пише:  
> > > >
> > > > On Sun, Nov 1, 2026 at 3:34 PM Demian Shulhan <demyansh@xxxxxxxxx> wrote:  
> > > > >
> > > > > I would prefer to keep only the scalar version of the fused copy+csum.
> > > > > It reads the data once instead of twice, and on real Armv9 hardware it
> > > > > cuts end-to-end cycles/byte for bulk TCP over WireGuard by 3-12%
> > > > > depending on the core (+6-8% throughput). It uses only general-purpose
> > > > > registers, so there is no FPSIMD state to save and it is safe in any
> > > > > context.
> > > > >
> > > > > If you are OK with the scalar path, I can send v2 with NEON dropped,
> > > > > the scalar path only, and the KUnit fix for the x86 zero-length case
> > > > > Jakub reported.
> > > > >  
> > > > The scalar path seems like generic C code and not restricted to arm64,
> > > > curious how that performs against existing (e.g. x86) assembly
> > > > optimized implementation.
> > > > saeed  
> >