Re: [PATCH v2] lib/crc: arm64: Add NEON PMULL+EOR3 crc64_nvme implementation

From: Shreesh Adiga

Date: Tue Nov 17 2026 - 01:50:31 EST


Hi Eric,

On Tue, Nov 17, 2026 at 5:31 AM Eric Biggers <ebiggers@xxxxxxxxxx> wrote:
> Was crc_kunit run? It already has a benchmark in it.
No, I have access to only one aarch64 device that is Android Mediatek
Dimensity 9400 SoC without root access.
Therefore I can't run crc_kunit on that, instead I have done userspace
comparisons for the same kernel with libbenchmark.
This is based on my earlier work for FFmpeg:
https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21651
I also have Corporate MacOS M5 and did check inside Ubuntu VM but that
is not accurate due to virtualization.

> The commit message should also clarify that this benefits ARMv8.2 and
> later processors with the sha3 extension, not just ARMv9. (E.g., Apple
> M1 is an example of an ARMv8.4 processor that benefits.)
Sure I will take care of it, but for now I am thinking of first
improving the existing PMULL to have 4x unrolling as per Demian's
suggestion.
Therefore I will revisit EOR3 addition later, will keep this in mind.

> > +#ifdef CONFIG_ARM64
> > +#define SHA3_TARGET __attribute__((target("arch=armv8.2-a+crypto+sha3")))
>
> Doesn't this break the build with gcc 8.1? The kernel supports gcc 8.1
> and later, but support for the sha3 target and intrinsics were added in
> gcc 9.1. Either this code will need to check the gcc version, or inline
> assembly will need to be used for eor3 like lib/raid/xor/ does.
Yes it seems to be unsupported even on GCC 8.2 and works on GCC 8.5
onwards, I wasn't aware of older toolchain support.
Will keep this in mind. Unfortunately compiler explorer is missing
ARM64 GCC 8.1, will check 8.2 support which hopefully should be
accurate.
Will adopt the existing code in lib/raid/xor itself which should make
it compatible with GCC 8.1.

> > + v0 = fold_single_round(v0, v2, fold_consts_2x);
> > + v1 = fold_single_round(v1, v3, fold_consts_2x);
> > + v0 = fold_single_round(v0, v1, fold_consts);
> > +
> > + while (len >= 16) {
> > + uint64x2_t t0 = vreinterpretq_u64_u8(vld1q_u8(p));
> > +
> > + v0 = fold_single_round(v0, t0, fold_consts);
> > + p += 16;
> > + len -= 16;
> > + }
> > +
> > + /* Multiply the 128-bit value by x^64 and reduce it back to 128 bits. */
> > + v0 = vextq_u64(v0, zero, 1) ^ pmull64_hi_lo(fold_consts, v0);
> > +
> > + /* Final Barrett reduction */
> > + uint64x2_t bconsts = vld1q_u64(bconsts_val);
> > + uint64x2_t final = pmull64(bconsts, v0);
> > +
> > + v0 = veor3q_u64(v0, vextq_u64(zero, final, 1),
> > + pmull64_hi_lo(bconsts, final));
> > +
> > + return vgetq_lane_u64(v0, 1);
>
> Note that this is missing several optimizations compared to the x86
> version (lib/crc/x86/crc-pclmul-template.S), including using
> fold_consts_2x for a tail of 'len & 32', efficiently handling lengths
> that aren't a multiple of 16 bytes, and reducing to the final CRC in two
> carryless multiplication instructions rather than three.
I was looking at the crc-pclmul-template.S code and I have a few queries.
In this aarch64 version we only dispatch to this SIMD routine only
when len>=128.
Since we have 128 bytes available, do we have to add the tail of `len
& 32` handling?
Handling len not a multiple of 16 bytes (partial block) I will add but
I am assuming that the dispatch will maintain 128 minimum len.

I was trying to understand your "reducing to the final CRC in two
carryless multiplications rather than three" statement.
>From going through the code in crc-pclmul-template.S:
.Lreduce_128bits_to_crc tracing the logic below this label, I see
three clmul operations:
_pclmulqdq CONSTS_XMM, LO64_TERMS, %xmm0, HI64_TERMS, %xmm1

_cond_vex movdqa, OFFSETOF_BARRETT_REDUCTION_CONSTS(CONSTS_PTR), CONSTS_XMM
_pclmulqdq CONSTS_XMM, HI64_TERMS, %xmm0, HI64_TERMS, %xmm1

.if LSB_CRC && \n == 64
_cond_vex punpcklqdq, %xmm1, %xmm2, %xmm2
_pclmulqdq CONSTS_XMM, LO64_TERMS, %xmm1, HI64_TERMS, %xmm1

I even checked the Intel Whitepaper and the reduction for 64-bit CRC
section "Final Reduction of 128-bits" also mentions 3 clmul.
One for using x^128 followed by xor, and then 2 more for final barrett
reduction with mu and poly.
Thus I am confused about accomplishing this with just 2 clmul. Could
you provide details on this?

Based on inputs I am thinking of only extending the PMULL version to
have unrolling with inline assembly for better codegen.
I will revisit EOR3 afterwards when the PMULL base version has
incorporated all the optimizations.
Mostly we can have always_inline function which can be instantiated
with/without EOR3 I think for approach.
Will submit new separate patch for just PMULL unrolling shortly in a
couple of days based on Demian's inputs.

Thanks,
Shreesh