Re: [PATCH RFC tip/core/rcu 0/16] Prototype RCU usable from idle, exception, offline

[Mathieu Desnoyers]

----- On Mar 13, 2020, at 11:42 AM, paulmck paulmck@xxxxxxxxxx wrote:

> On Fri, Mar 13, 2020 at 03:41:46PM +0100, Frederic Weisbecker wrote:
>> On Thu, Mar 12, 2020 at 11:16:18AM -0700, Paul E. McKenney wrote:
>> > Hello!
>> > 
>> > This series provides two variants of Tasks RCU, a rude variant inspired
>> > by Steven Rostedt's use of schedule_on_each_cpu(), and a tracing variant
>> > requested by the BPF folks and perhaps also of use for other tracing
>> > use cases.
>> > 
>> > The tracing variant has explicit read-side markers to permit finite grace
>> > periods even given in-kernel loops in PREEMPT=n builds It also protects
>> > code in the idle loop, on exception entry/exit paths, and on the various
>> > CPU-hotplug online/offline code paths, thus having protection properties
>> > similar to SRCU.  However, unlike SRCU, this variant avoids expensive
>> > instructions in the read-side primitives, thus having read-side overhead
>> > similar to that of preemptible RCU.
>> > 
>> > There are of course downsides.  The grace-period code can send IPIs to
>> > CPUs, even when those CPUs are in the idle loop or in nohz_full userspace.
>> > It is necessary to scan the full tasklist, much as for Tasks RCU.  There
>> > is a single callback queue guarded by a single lock, again, much as for
>> > Tasks RCU.  If needed, these downsides can be at least partially remedied
>> 
>> So what we trade to fix the issues we are having with tracing against extended
>> grace periods, we lose in CPU isolation. That worries me a bit as tracing can
>> be thoroughly used with nohz_full and CPU isolation.
> 
> First, disturbing nohz_full CPUs can be avoided by the sysadm simply
> refusing to remove tracepoints while sensitive applications are running
> on nohz_full CPUs.

I doubt this approach will survive real-life.

> 
> Second, for non-CPU-bound real-time programs with mostly-idle CPUs,
> I should be able to decrease the likelihood of sending IPIs pretty much
> to zero.
> 
> Or am I missing something here?

I would recommend considering the following alternative for this tracing-rcu
flavor:

- For all CPUs which are not nohz_full:
  - Implement fast RCU read-side which only requires compiler barriers,
  - Use IPIs to each of those CPUs when doing a grace period.

- For all nohz_full CPUS:
  - Dynamically detect CPUs which are nohz_full,
  - Implement slower RCU read-side with memory barriers,
  - No need to issue any IPI to those CPUs when doing the grace period.

This should cover all use-cases: staying fast for the common case, without
disturbing RT workloads.

Thoughts ?

Thanks,

Mathieu




-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com