Re: [PATCH 0/2] arm64: csum: Add fused copy and Internet checksum

[Demian Shulhan]

> The scalar path seems like generic C code and not restricted to arm64,
> curious how that performs against existing (e.g. x86) assembly
> optimized implementation.

Hi David, Saeed,

I tested David's sum64() loop, with the store added, on x86-64 and arm64.
It is portable and correct on both, but no single C form was fast on
both:

- x86-64, vs csum-copy_64.S (GCC 15, Clang 19-22): with dst alignment
  and 64 bytes per pass it reaches 0.89-0.94 of the asm. The form costs
  two ALU ops per word (add + adc $0) against one adc in the asm, so even
  a hand-written asm version of it stays at 0.94-0.98.
- arm64, vs the scalar loop from my series (four accumulators,
  res + (res < data)): the same 64-bit loop is 0.90x, while the same
  loop on 128-bit val/sum with 2-4 independent lanes is 1.19-1.24x. On
  x86 those 128-bit lanes drop to 0.40-0.52 of the asm with GCC.

So I'll keep it in arch/arm64, and x86 keeps its asm. In the next
version I'll move the loop to 128-bit val/sum with independent lanes,
the same way arm64's do_csum() already works on __uint128_t.

Demian

вт, 3 лист. 2026 р. о 23:38 David Laight <david.laight.linux@xxxxxxxxx> пише:
>
> On Tue, 3 Nov 2026 16:47:29 +0100
> Demian Shulhan <demyansh@xxxxxxxxx> wrote:
>
> > > .L4:
> > >         ldr     x1, [x0, x4]
> > >         adds    x2, x2, x1
> > >         cinc    x3, x3, cs      # adc $0,%rxx on x86
> > >         adds    x4, x4, 8
> > >         bne     .L4
> > Agreed, that is a nice loop for the sum-only case, one dependent op on
> > the sum chain and no builtin needed.
> > For the copy case I tried the same form on x86 (Zen 3, GCC 15, Clang
> > 19-22, -O2) with a store added, against csum-copy_64.S, as posted it ran
> > at 0.85x of the asm with GCC and 0.73-0.76x with Clang (which rewrites
> > it into setb/adc), and ~0.8x unrolled 2x/4x - a bit slower than the
> > posted 4-accumulator loop (0.88-0.95x).
>
> I just added *(u64 *)(dest + count) = val to one of my loops.
> gcc just replaces the 'add from memory' with mov/mov/add.
>
> I'm now confused though, I seem to be getting different output from earlier.
> With -Os (which I didn't think I was passing) clang is refusing to do the
> 'offset from end of buffer' loop.
> With -O2 it bloats everything by unrolling 4 times and adding lots of
> extra instructions.
> (AFAICT the optimiser has a completely incorrect cpu model.)
>
> If the compiler doesn't generate the expected code it is all pointless.
> The xor parity code loop had to be in the asm block to stop the compiler
> changing it to a much slower form.
>
> For the checksum code I have managed to work out how many bytes/clock
> the code was generating and getting it to (reasonably) match the expected
> values.
> But you have to use relatively short buffers to avoid D-cache effects.
>
> David
>
> >
> > пн, 2 лист. 2026 р. о 17:14 David Laight <david.laight.linux@xxxxxxxxx> пише:
> >
> >
> > >
> > > On Mon, 2 Nov 2026 16:14:06 +0100
> > > Demian Shulhan <demyansh@xxxxxxxxx> wrote:
> > >
> > > > > The scalar path seems like generic C code and not restricted to arm64,
> > > > > curious how that performs against existing (e.g. x86) assembly
> > > > > optimized implementation.
> > > > Depends. I think __builtin_addcll() is needed to get roughly the same
> > > > performance as the asm version. In my tests GCC on arm64 turns it
> > > > into adds + cset instead of adcs like on x86.
> > > > But I will look at it, thanks!
> > >
> > > The C version I posted earlier has the loop:
> > >     buf += count;
> > >     count = -count;
> > >     sum = carry = 0;
> > >     do {
> > >         val = *(u64 *)(buf + count);
> > >         sum += val;
> > >         carry += sum < val;
> > >     } while ((count += 8));
> > >
> > > for aarch64 you get:
> > > .L4:
> > >         ldr     x1, [x0, x4]
> > >         adds    x2, x2, x1
> > >         cinc    x3, x3, cs      # adc $0,%rxx on x86
> > >         adds    x4, x4, 8
> > >         bne     .L4
> > >
> > > I suspect that is close to optimal - especially for fast cpu with
> > > out of order execution.
> > >
> > > https://godbolt.org/z/eso79K7Wq
> > >
> > > David
> > >
> > > >
> > > > пн, 2 лист. 2026 р. о 12:44 saeed bishara <saeed.bishara.os@xxxxxxxxx> пише:
> > > > >
> > > > > On Sun, Nov 1, 2026 at 3:34 PM Demian Shulhan <demyansh@xxxxxxxxx> wrote:
> > > > > >
> > > > > > I would prefer to keep only the scalar version of the fused copy+csum.
> > > > > > It reads the data once instead of twice, and on real Armv9 hardware it
> > > > > > cuts end-to-end cycles/byte for bulk TCP over WireGuard by 3-12%
> > > > > > depending on the core (+6-8% throughput). It uses only general-purpose
> > > > > > registers, so there is no FPSIMD state to save and it is safe in any
> > > > > > context.
> > > > > >
> > > > > > If you are OK with the scalar path, I can send v2 with NEON dropped,
> > > > > > the scalar path only, and the KUnit fix for the x86 zero-length case
> > > > > > Jakub reported.
> > > > > >
> > > > > The scalar path seems like generic C code and not restricted to arm64,
> > > > > curious how that performs against existing (e.g. x86) assembly
> > > > > optimized implementation.
> > > > > saeed
> > >
>