Re: [PATCH v4 1/1] powerpc: enable dynamic preemption
From: Paul E. McKenney
Date: Mon Jul 27 2026 - 12:24:48 EST
On Mon, Jul 27, 2026 at 12:35:32PM +0200, Christophe Leroy (CS GROUP) wrote:
> Hi Shrikanth,
>
> Le 27/07/2026 à 12:28, Shrikanth Hegde a écrit :
> > Hi Jirka,
> > Please avoid top-posting.
> >
> > On 7/27/26 3:43 PM, Jirka Hladky wrote:
> > > Hi Shrikanth,
> > >
> > > Thanks for the quick response and for looping in Paul.
> > >
> > > > This is true only if user selected PREEMPT_DYNAMIC option i think.
> > >
> > > Yes, exactly. The issue is that CONFIG_PREEMPT_DYNAMIC=y has been in
> > > the Fedora/RHEL kernel config since kernel 5.16 (2021). It was
> > > silently ignored on ppc64le until your 6.16 patch added
> > > HAVE_PREEMPT_DYNAMIC_KEY, so this is the first time PREEMPT_RCU
> > > actually takes effect for all Fedora/RHEL ppc64le users.
> > >
> > > > Does your preemption mode remain the same in two cases?
> > >
> > > Verified. On 6.15-rc6 (before PREEMPT_DYNAMIC takes effect):
> > >
> > > CONFIG_PREEMPT_VOLUNTARY=y
> > > (no CONFIG_PREEMPT_DYNAMIC, no /sys/kernel/debug/sched/preempt)
> > >
> > > On both 7.1 and 7.2-rc4 (with PREEMPT_DYNAMIC active):
> > >
> > > CONFIG_PREEMPT_LAZY=y
> > > CONFIG_PREEMPT_DYNAMIC=y
> > > CONFIG_PREEMPT_RCU=y
> > > cat /sys/kernel/debug/sched/preempt: "full (lazy)"
> > >
> > > So the runtime preemption mode is the same on 7.1 and 7.2. The
> > > difference between 6.15 and 7.1+ is that the preemption model changed
> > > from voluntary (static) to full/lazy (dynamic), which is what pulls
> > > in PREEMPT_RCU.
> >
> > That means comparison is between preempt=voluntary vs preempt=lazy.
>
> As far as I can see CONFIG_PREEMPT_LAZY doesn't pull CONFIG_PREEMPT_RCU.
> CONFIG_PREEMPT_DYNAMIC does:
That is correct.
Now, PREEMPT_LAZY does add preempt_disable() to rcu_read_lock()
and preempt_enable() to rcu_read_unlock(), compared to PREEMPT_NONE.
So there might be some performance impact. But hopefully not quite as
much as an isync or lwsync instruction.
OK, OK, PREEMPT_NONE also has preempt_disable() in rcu_read_lock() and
preempt_enable() in rcu_read_unlock(). The real difference is that in
PREEMPT_NONE, both preempt_disable() and preempt_enable() are no-ops.
In contrast, in PREEMPT_LAZY these two execute real code.
Thanx, Paul
> config PREEMPT_RCU
> bool
> default y if (PREEMPT || PREEMPT_RT || PREEMPT_DYNAMIC)
> select TREE_RCU
> help
> This option selects the RCU implementation that is
> designed for very large SMP systems with hundreds or
> thousands of CPUs, but for which real-time response
> is also required. It also scales down nicely to
> smaller systems.
>
> Select this option if you are unsure.
>
>
> Christophe
>
> >
> > If you make it full preemption in 6.15 you will likely see similar data
> > as 7.1+.
> > Only on 7.0/7.1 there is force switch to lazy/full. Can you give that a
> > try?
> >
> > If it shows same data, that implies the regression is mainly due to change
> > of preemption modes, rather than the static key stuff.
> >
> > >
> > > > Weak memory model would need barriers irrespective of
> > > > HAVE_PREEMPT_DYNAMIC_CALL or HAVE_PREEMPT_DYNAMIC_KEY.
> > > > Static key too is expected to minimal cost.
> > >
> > > You're right, I should clarify -- the overhead is not in the static
> > > key mechanism itself. The cost comes from __rcu_read_lock() and
> > > __rcu_read_unlock() which are called when CONFIG_PREEMPT_RCU=y.
> > > These need real memory barriers (lwsync/isync) on ppc64le regardless
> > > of whether the dynamic mechanism uses keys or calls.
> > >
> > > The real question is: why does enabling PREEMPT_RCU cost ~33% on
> > > ppc64le but only ~3% on x86_64? The answer is that x86_64's TSO
> > > memory model makes the barriers in __rcu_read_lock/__rcu_read_unlock
> > > essentially free, while ppc64le's weak ordering requires explicit
> > > lwsync/isync instructions, which are expensive when called thousands
> > > of times per second in the SELinux AVC hot path.
> >
> > Plus, it may call schedule in lazy/pull preemption.
> >
> > >
> > > I also have new data from SELinux isolation testing that helps
> > > quantify this. On kernel 7.1 (which already has PREEMPT_RCU=y):
> > >
> > > SELinux mode kill bogo-ops/sec vs enforcing
> > > ------------ ----------------- ------------
> > > Enforcing 69,107 baseline
> > > Permissive 70,248 +1.6%
> > > Disabled 93,566 +35.4%
> > >
> > > Permissive ~ enforcing confirms the overhead is not in SELinux policy
> > > evaluation. Disabling SELinux removes the rcu_read_lock/unlock call
> > > sites in the AVC path and recovers most of the performance -- but
> > > still leaves a ~13% gap vs 6.12 (no PREEMPT_RCU), which is the base
> > > cost of PREEMPT_RCU in the non-SELinux parts of the kill() path.
> > >
> >
> > This seems strange. How come rcu lock/unlock depends on SELinux policy?
> > One should call rcu lock/unlock if they are working with rcu updated
> > fields.
> >
> > Does the policy change itself protected with rcu lock/unlock?
> > I will check it up.
> >
> > > So the core issue is: on ppc64le, CONFIG_PREEMPT_RCU makes
> > > rcu_read_lock/unlock significantly more expensive, and the kill()
> > > syscall path hits them very heavily through SELinux AVC lookups.
> > >
> > > Thanks,
> > > Jirka
> > >
> > > On Mon, Jul 27, 2026 at 6:19 AM Shrikanth Hegde
> > > <sshegde@xxxxxxxxxxxxx> wrote:
> > > >
> > > > +cc paul for any further/RCU insights.
> > > >
> > > > On 7/27/26 12:03 AM, Jirka Hladky wrote:
> > > > > Hi Shrikanth, Christophe,
> > > > >
> > > >
> > > > Hi Jirka, thanks for the report.
> > > >
> > > > > I'm seeing a significant performance regression on
> > > > > ppc64le after this
> > > > > patch landed in 6.16, caused by CONFIG_PREEMPT_RCU becoming active
> > > > > once HAVE_PREEMPT_DYNAMIC_KEY is selected.
> > > > >
> > > >
> > > > This is true only if user selected PREEMPT_DYNAMIC option i think.
> > > >
> > > > config PREEMPT_RCU
> > > > bool
> > > > default y if (PREEMPT || PREEMPT_RT || PREEMPT_DYNAMIC)
> > > > select TREE_RCU
> > > >
> > > >
> > > > > Benchmark: stress-ng kill stressor (tight kill() syscall loop),
> > > > > single thread, POWER10 LPAR (8 vCPUs, 1 core SMT-8).
> > > > >
> > > > > Bisected across Fedora ELN kernel builds on ppc64le:
> > > > >
> > > > > kernel CONFIG_PREEMPT_RCU kill bogo-ops/sec
> > > > > --- 6.15-rc6 (eln148) no 103,207
> > > > > 6.16 (eln150) yes 70,281 (-32%)
> > > > > 6.18 (eln154) yes 72,552 (-30%)
> > > > >
> > > >
> > > > Does your preemption mode remain the same in two cases?
> > > >
> > > > > For comparison, x86_64 (AMD EPYC 9355P) with the same config change
> > > > > shows only a 2.8% regression:
> > > > >
> > > > > 6.12 x86_64 37,436
> > > > > 7.2 x86_64 36,392 (-2.8%)
> > > > >
> > > > > perf report shows the overhead comes from
> > > > > rcu_read_lock/unlock in the
> > > > > SELinux AVC path (check_kill_permission -> security_task_kill ->
> > > > > selinux_task_kill -> avc_has_perm -> avc_lookup):
> > > > >
> > > > > Function 6.15 (no PREEMPT_RCU) 6.16 (PREEMPT_RCU)
> > > > > --- avc_lookup 15.23% 24.79%
> > > > > __rcu_read_lock ~0% 4.52%
> > > > > __rcu_read_unlock ~0% 4.17%
> > > > > selinux_task_kill 6.23% 7.35%
> > > > > audit_signal_info* 0.94% 3.59%
> > > > >
> > > > > On x86_64, rcu_read_lock/unlock are cheap thanks to static calls
> > > > > (HAVE_PREEMPT_DYNAMIC_CALL). On ppc64le with the KEY-based
> > > > > implementation, the weak memory model requires real barriers
> > > > > (lwsync/isync) making each RCU read-side critical section
> > > > > significantly more expensive.
> > > >
> > > > Weak memory model would need barriers irrespective of
> > > > HAVE_PREEMPT_DYNAMIC_CALL
> > > > or HAVE_PREEMPT_DYNAMIC_KEY. That's my assumption. I will look
> > > > more into it. Also i don't know much about PREEMPT_RCU. So might
> > > > take a while.
> > > >
> > > > >
> > > > > This aligns with Christophe's earlier review comment that
> > > > > HAVE_PREEMPT_DYNAMIC_CALL should be more performant. Would
> > > > > implementing static calls for ppc64 be feasible to close this gap?
> > > > >
> > > >
> > > > Static key too is expected to minimal cost. There maybe more into this.
> > > >
> > > > > Test details:
> > > > > - Machine: IBM POWER10 (pvr 0080 0200), pHyp virtualization
> > > > > - stress-ng 0.21.03, gcc 14.3.1, glibc 2.39
> > > > > - Tuned profile: virtual-guest
> > > > > - SELinux: enforcing (permissive recovers only ~7%)
> > > > >
> > > > > Happy to run additional tests if needed.
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>