Re: [PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[Rosen Penev]

On Mon, Oct 12, 2026 at 11:38 AM Christophe Leroy (CS GROUP)
<chleroy@xxxxxxxxxx> wrote:
>
>
>
> Le 12/10/2026 à 20:28, Rosen Penev a écrit :
> > On Mon, Oct 12, 2026 at 11:18 AM Christophe Leroy (CS GROUP)
> > <chleroy@xxxxxxxxxx> wrote:
> >>
> >>
> >>
> >> Le 12/10/2026 à 20:00, Rosen Penev a écrit :
> >>> __csum_partial() walks the buffer linearly and reads each cache line
> >>> exactly once. On cores without a hardware prefetcher, such as the 464FP
> >>> in the APM82181, every new line is therefore a demand miss that stalls
> >>> the loop. Packets received by non-coherent DMA are never in the cache,
> >>> so this is the common case for GRO checksum validation: when routing
> >>> TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
> >>> of all cycles and is the top entry in the profile.
> >>>
> >>> Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
> >>> dcbt is only a hint and never faults, so prefetching past the end of
> >>> the buffer is harmless. Lines that are already cached, for example when
> >>> checksumming data the CPU just wrote, are left untouched.
> >>>
> >>> Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
> >>> routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
> >>> was toggled at runtime, in the same boot:
> >>>
> >>>                 LAN->WAN               WAN->LAN
> >>>                 Mbit/s  csum cycles    Mbit/s  csum cycles
> >>>     before      495     30.5%          439     27.2%
> >>>     after       599     15.3%          520     13.7%
> >>>
> >>> That is 58% less time spent in __csum_partial() per byte.
> >>>
> >>> The prefetch distance was chosen with an earlier, slower driver
> >>> configuration (median of 3, Mbit/s):
> >>>
> >>>     distance    LAN->WAN   WAN->LAN
> >>>     none        370        341
> >>>     64 bytes    430        381
> >>>     96 bytes    438        397
> >>>     128 bytes   438        385
> >>>     160 bytes   434        386
> >>>     256 bytes   423        383
> >>>
> >>> Prefetching the first lines before entering the loop made no
> >>> measurable difference and is not done.
> >>>
> >>> Assisted-by: LLM
> >>> Signed-off-by: Rosen Penev <rosenp@xxxxxxxxx>
> >>
> >> csum_partial_copy_generic() does the same, so your patch makes sense.
> >>
> >> Wondering if you should use MAX_COPY_PREFETCH instead of hardcoding value 4.
> > li      r9,MAX_COPY_PREFETCH*L1_CACHE_BYTES /* prefetch distance */
> >
> > LLM comment:
> >
> > What changes per platform: MAX_COPY_PREFETCH is 4 on every 32-bit
> > config except 8xx, where it is 1. So 44x, 47x and e500mc behave
> > exactly as before; on the 44x (canyonlands) build the instruction is
> > still li r9, 128. The one difference is 8xx, which now prefetches 16
> > bytes ahead instead of 64. That fits how the copy routine already
> > treats 8xx, but I didn't benchmark it.
> >
> > My comment:
> > I don't have an 8xx device so I don't know if this works properly there or not.
>
> I have an 885 and might do the test one day, but I won't have time
> during the coming weeks. Nevertheless I have the feeling it makes sense
> to implement something based on MAX_COPY_PREFETCH to resemble
> csum_partial_copy_generic()
I got a user to test this setup with MAX_COPY_PREFETCH

Using Sophos RED 15w, right after sysupgrade -n, got an iperf3 -c
result of ~633Mb/s with the suggested change, vs. ~583Mb/s with the
latest snapshot build from the web site, ie. without this PR.
>
> >>
> >> Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@xxxxxxxxxx>
> > I'll keep this in v3.
> >>
> >>> ---
> >>>    v2: reword commit message and retest numbers.
> >>>    arch/powerpc/lib/checksum_32.S | 4 +++-
> >>>    1 file changed, 3 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
> >>> index cd00b9bdd772..5a14d45c7b9d 100644
> >>> --- a/arch/powerpc/lib/checksum_32.S
> >>> +++ b/arch/powerpc/lib/checksum_32.S
> >>> @@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
> >>>        bdnz    2b
> >>>    21: srwi.   r6,r4,4         /* # blocks of 4 words to do */
> >>>        beq     3f
> >>> +     li      r9,4*L1_CACHE_BYTES     /* prefetch distance */
> >>>        lwz     r0,4(r3)
> >>>        mtctr   r6
> >>>        lwz     r6,8(r3)
> >>> @@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
> >>>        lwzu    r8,16(r3)
> >>>        adde    r5,r5,r7
> >>>        bdz     23f
> >>> -22:  lwz     r0,4(r3)
> >>> +22:  dcbt    r3,r9
> >>> +     lwz     r0,4(r3)
> >>>        adde    r5,r5,r8
> >>>        lwz     r6,8(r3)
> >>>        adde    r5,r5,r0
> >>
>