Re: [PATCH v4 1/1] powerpc: enable dynamic preemption

[Shrikanth Hegde]

Hi Jirka, Paul,

On 7/28/26 7:49 PM, Jirka Hladky wrote:
> On Tue, Jul 28, 2026 at 7:12 AM Shrikanth Hegde <sshegde@xxxxxxxxxxxxx> wrote:
> > No. It doesn't confirm. I would recommend you do that case to find out
> > the cost of PREEMPT_RCU alone.
> >
> > Config A: CONFIG_PREEMPT_VOLUNTARY=y (CONFIG_PREEMPT_DYNAMIC=n)
> > Config B: CONFIG_PREEMPT_VOLUNTARY=y (CONFIG_PREEMPT_DYNAMIC=y)
>
> Done. Built two kernels from 6.18 upstream source on the same POWER10
> machine. Both are voluntary preemption, differing only in
> PREEMPT_DYNAMIC (and thus PREEMPT_RCU):
>
> Config A: PREEMPT_VOLUNTARY=y, PREEMPT_DYNAMIC=n (no PREEMPT_RCU)
> Config B: PREEMPT_VOLUNTARY=y, PREEMPT_DYNAMIC=y (PREEMPT_RCU=y)
>
> Config B runtime mode confirmed as voluntary:
>    cat /sys/kernel/debug/sched/preempt: "none (voluntary) full lazy"
>
> Results (stress-ng --kill 1 -t 23, SELinux enforcing, mean of 3 runs):
>
> Kernel                  PREEMPT_RCU   kill bogo-ops/sec   stddev
> ----------------------  -----------   -----------------   ------
> 6.18.0-vol-nodynamic    no            108,836             19
> 6.18.0-vol-dynamic      yes            68,197             1,146
> Delta                                 -37.3%
>
> PREEMPT_RCU alone costs 37.3% on ppc64le with zero preemption mode
> change. Both kernels run voluntary preemption.

I tried to repro using "taskset -c 0-7 stress-ng --kill 1 -t 23 --metrics"
since system had more than 1 core.
I did observe close to 25% regression with Config B

Config A: PREEMPT_VOLUNTARY=y, PREEMPT_DYNAMIC=n
Config B: PREEMPT_VOLUNTARY=y, PREEMPT_DYNAMIC=y

I spent some more time debugging it as it was bugging me for few reasons.

1. There is barrier even when CONFIG_PREEMPTION=n.
See include/linux/preempt.h.
#define preempt_disable()                       barrier()
#define preempt_enable()                        barrier()
   So i was doubting all the cost is due to that.

2. Seeing the stress-ng --kill 1 -t 23, it spawns 3 threads IIUC.
   two will be long running. Given there are 8 CPUs in the test,
   preemption should be minimal. (This is only a guess)

3. Though rcu_lock/unlock shows up only in one case, it could be just
   that they are inline in PREEMPT_DYNAMIC=n. They still do barriers.

A did below hacks to see where the cost is coming from.

===================================
First thing i did was to remove PREEMPT_RCU for PREEMPT_DYNAMIC=y.

diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig
index 4d9b21f69eaa..1d67060b2261 100644
--- a/kernel/rcu/Kconfig
+++ b/kernel/rcu/Kconfig
@@ -18,7 +18,7 @@ config TREE_RCU
 
 config PREEMPT_RCU
        bool
-       default y if (PREEMPT || PREEMPT_RT || PREEMPT_DYNAMIC)
+       default y if (PREEMPT || PREEMPT_RT)
        select TREE_RCU
        help
          This option selects the RCU implementation that is

This improves it by 5%. But doesn't cover the whole regression.
============================================================================

So PREEMPT_RCU is not the main reason. Now what else PREEMPT_DYNAMIC
adds,
 
1. PREEMPT_DYNAMIC select CONFIG_PREEMPTION and that selects PREEMPT_COUNT.
   That turns every preempt_enable/disable to dec/inc the preempt count.
   Even though it isn't finally used for voluntary preemption.
   This can have overhead on any RISC architecture if all the workload does is
   hitting that path, without doing any meaningful work.

2. Static key overheads.

++++++++++++++++++++++++++++++++++++++
For 1, first thought came to my mind was is it due to preempt_count
being part of thread_info. Tried to move it PACA as it is more efficient(atleast
for 64 bit). Even with that numbers didn;t improve much. Then i thought,
is it purely due to preempt count itself. So did below,

diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index da326800c1c9..d00c600dc071 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -117,11 +117,11 @@ config PREEMPT_RT_NEEDS_BH_LOCK
          the old synchronized behaviour.
 
 config PREEMPTION
-       bool
-       select PREEMPT_COUNT
+       bool
+       select PREEMPT_COUNT if !PREEMPT_VOLUNTARY
 
That is able to recover all the regression. You can give it a try if you want.
But that's not a solution as dynamic preemption just won't work.

++++++++++++++++++++++++
For 2, Sadly, i lost the system to try any hacks for static key overheads.
But given above, i don't expect it to be a major one.

==============================================================================

From the above, it is quite evident that main cost is cost of preempt_disable/preempt_enable
due to inc/dec of preempt count itself. This is due to nature of this workload which
probably hits just lock/unlock likely, and no other work outside of it.

The cost of preempt_count is likely un-avoidable with DYNAMIC_PREEMPTION for
all RISC architecture. I think CISC cost is minimal since it can update memory and
folding of need_resched bit into the preempt count. Both are not doable in RISC.

Also given that after 7.0, all major archs including powerpc, can have only full/lazy
preemption. None/voluntary are no longer possible. So making any sort of optimization for
preempt count is not useful since it has to be there for full/lazy.

Also, evaluate it with any real life workloads, IIRC i have run hackbench, schbench,
daytrader(db2 workload) this cost wasn't visible.

> > That's true for all archs. Please check your preemption mode in your
> > x86 experiment.
>
> Checked. On x86_64 (AMD EPYC 7313):
>
> Kernel               PREEMPT_RCU   Runtime mode        kill bogo-ops/sec
> -------------------  -----------   ------------------  -----------------
> 6.12.0-211 (el10)    yes           none (voluntary)    37,436
> 7.2.0-rc4 (eln158)   yes           full (lazy)         36,392 (-2.8%)
>
> Both x86 kernels already had PREEMPT_RCU=y, so the x86 comparison
> only shows the voluntary->lazy mode change cost (~3%), not the
> PREEMPT_RCU enablement cost.