Re: [PATCH] lib/percpu_counter: allow concurrent sums with an rwlock

From: Andrew Morton

Date: Sun Nov 15 2026 - 22:16:40 EST


On Mon, 16 Nov 2026 09:56:55 +0800 Jiebin Sun <jiebin.sun@xxxxxxxxx> wrote:

> __percpu_counter_sum() holds a raw spinlock while reading the global
> count and per-CPU deltas, serializing concurrent sums of the same
> counter. Since a sum changes neither the global count nor the per-CPU
> deltas, use rwlock_t on !PREEMPT_RT to allow concurrent sums with a
> read lock. Global updates take the write lock. The per-CPU fast path
> remains lockless, so sums keep their existing snapshot semantics.
>
> stress-ng dentry reaches this path through ext4_statfs():
>
> statfs -> ext4_statfs -> __percpu_counter_sum
> -> native_queued_spin_lock_slowpath
>
> Keep the raw spinlock on PREEMPT_RT, where rwlock_t can sleep.
> Global counter updates can run in atomic context and must not sleep.
>
> stress-ng dentry test average throughput change:
>
> +------------------------+------------+--------------------+
> | Case | Unit | Average change (%) |
> +------------------------+------------+--------------------+
> | Dentry, 64 workers | bogo ops/s | +47.34 |
> +------------------------+------------+--------------------+
>
> Dentry ran on ext4 with 64 workers pinned to physical CPUs:
>
> stress-ng --dentry 64 --dentries 4096 --timeout 20s

lol, this is a fun patch - it reminds me of the old days. All those
mission-critical workloads which sit in a loop running statfs will love
it!

I had $sidekick do a scan and it told me "the only plausible routine
win is dirty throttling: with low dirty limits, concurrent writers sum
the same counters. ext4 and XFS sum near ENOSPC, where updates also
contend. BFQ's sums are debug-only and serialized. Most other callers
report statistics". You might want to add something like this to
changelog.

Also mention that percpu_counter_sum_positive callers will benefit.

> --- a/include/linux/percpu_counter.h
> +++ b/include/linux/percpu_counter.h
> @@ -20,7 +20,11 @@
> #ifdef CONFIG_SMP
>
> struct percpu_counter {
> +#ifdef CONFIG_PREEMPT_RT

Might be worth a wee comment telling readers why RT is special?

> raw_spinlock_t lock;
> +#else
> + rwlock_t lock; /* shared for sums, exclusive for updates */
> +#endif
> s64 count;
> #ifdef CONFIG_HOTPLUG_CPU
> struct list_head list; /* All percpu_counters are on a list */
>
> --- a/lib/percpu_counter.c
> +++ b/lib/percpu_counter.c
> @@ -15,6 +15,46 @@ static LIST_HEAD(percpu_counters);
> static DEFINE_SPINLOCK(percpu_counters_lock);
> #endif
>
> +/*
> + * Non-RT sums share the lock; global updates take it exclusively.
> + * Keep raw spinlocks on RT: rwlocks may sleep and their irqsave variants
> + * do not disable interrupts. Updates run in atomic contexts and need
> + * IRQ exclusion for CPU-local read-modify-write sequences.
> + */
> +#ifdef CONFIG_PREEMPT_RT
> +#define fbc_lock(fbc) raw_spin_lock(&(fbc)->lock)
> +#define fbc_unlock(fbc) raw_spin_unlock(&(fbc)->lock)
> +#define fbc_lock_irqsave(fbc, f) raw_spin_lock_irqsave(&(fbc)->lock, f)
> +#define fbc_unlock_irqrestore(fbc, f) raw_spin_unlock_irqrestore(&(fbc)->lock, f)
> +#define fbc_read_lock_irqsave fbc_lock_irqsave
> +#define fbc_read_unlock_irqrestore fbc_unlock_irqrestore
> +#else
> +#define fbc_lock(fbc) write_lock(&(fbc)->lock)
> +#define fbc_unlock(fbc) write_unlock(&(fbc)->lock)
> +#define fbc_lock_irqsave(fbc, f) write_lock_irqsave(&(fbc)->lock, f)
> +#define fbc_unlock_irqrestore(fbc, f) write_unlock_irqrestore(&(fbc)->lock, f)
> +#define fbc_read_lock_irqsave(fbc, f) read_lock_irqsave(&(fbc)->lock, f)
> +#define fbc_read_unlock_irqrestore(fbc, f) read_unlock_irqrestore(&(fbc)->lock, f)
> +#endif

Please, no!

https://www.google.com/search?q=why+does+Andrew+Morton+prefer+static+inlines+over+macros

lots of good stuff there ;)