Re: [PATCH] lib/crc: Add 4-way folding in NEON crc64_nvme implementation
From: Shreesh Adiga
Date: Wed Nov 18 2026 - 11:48:54 EST
On Wed, Nov 18, 2026 at 9:59 PM Ard Biesheuvel <ardb@xxxxxxxxxx> wrote:
>
>
> On Wed, 18 Nov 2026, at 17:01, Shreesh Adiga wrote:
> > Use 4-way folding in the shared NEON crc64_nvme for AArch64
> > as well as AArch32. This improves performance by about 2x for 512 byte
> > data size on Cortex-X925.
> >
> > Add similar optimizations from x86 implementation like reducing
> > the 4x to 2x and doing a tail (len >= 32) 2x fold as well as partial
> > block handling to process lengths which are not multiple of 16 bytes.
> >
> > The partial block stage is enabled only for AArch64 as it needs the
> > 128-bit TBL instruction not available in AArch32.
> > Modified the loop dispatch on AArch64 to allow any len >= 128 rather
> > than just multiple of 16 bytes.
> >
> > Modified the pmull intrinsics from crc64-neon.h to use inline
> > assembly as the intrinsic usage was causing the compiler to
> > generate the fold constants via mov+movk+fmov/dup which was about 10
> > instructions for 16-byte constant instead of expected adrp+ld1.
> >
>
> Why is that better?
Based on my counting, 10 instructions (mov + movk + movk + movk + dup;
mov + movk + movk + movk + fmov) for loading a single 16-byte fold
constant = 40 bytes. Whereas now it is 2 instructions = 8 bytes + 16
bytes rodata.
Since we have 4 such fold constants pairs (4x, 2x, 1x, barret) it
reduces the instruction count significantly as well.
So lesser read-only memory usage. Also for in-order cores I assume
that the instruction sequence has serial dependency as well.
Previous discussion with Demian who had highlighted this point and
suggested to use inline asm for PMULL rather than intrinsic:
"Scalar vmull_p64() makes compilers materialize the constants via
mov/movk + fmov (42/97 insns on Clang)."
Hopefully the above reasoning is sufficient justification for this change.
Thanks,
Shreesh