Re: [PATCH-next 00/23] cgroup/cpuset: Enable runtime update of nohz_full and managed_irq CPUs

[Jing Wu]

Since we offered to take the RCU/nocb piece, here is a concrete shape to
shoot at.

Constraint recap (kernel/rcu/tree_nocb.h): rcu_nocb_cpu_{offload,deoffload}()
require the CPU offline (they reject when cpu_online()) under cpus_read_lock +
nocb_mutex, and per Frederic the toggle really needs CPUHP_AP_IDLE_DEAD because
callbacks can still be enqueued before that.  Reaching IDLE_DEAD pays the
stop_machine cost that disturbs other isolated partitions.

Observation: among the kernel-noise types, only RCU needs that deep offline;
tick, managed_irq and the watchdog reconfigure through the existing online-side
callbacks.  So bundling RCU under HK_TYPE_KERNEL_NOISE makes every isolation
change pay the RCU cost.

Proposal:

  1. Split HK_TYPE_RCU out of HK_TYPE_KERNEL_NOISE into its own
     runtime-mutable type, plus an immutable HK_TYPE_RCU_BOOT snapshot.
     HK_TYPE_KERNEL_NOISE keeps tick/timer/misc/wq.

  2. Gate nocb changes behind an explicit, per-partition cpuset opt-in.
     Default off: partition changes never touch rcu_nocb_mask, so no spike is
     inflicted on other partitions.  On: the isolated CPUs join/leave the nocb
     set, paying the offline cost once at setup/teardown.  This is Waiman's
     admin-choice knob.

  3. Do the toggle inside the serialized hotplug transition at
     CPUHP_AP_IDLE_DEAD, rather than the remove_cpu() -> toggle -> add_cpu()
     dance our v3 prototype used, which raced with concurrent hotplug.

Open questions:

  - Frederic: is CPUHP_AP_IDLE_DEAD the right hook, and is a new cpuhp callback
    there acceptable versus the current post-offline API?

  - admin pre-offlines the target CPUs (your earlier suggestion) versus the
    cpuset code cycling them via Thomas's per-CPU down/up primitive?

  - does the "only RCU needs the deep offline" assumption hold for tick,
    managed_irq and the watchdog?

If this shape sounds acceptable we will build it on top of your CPU down/up
primitives.

Thanks,
Jing Wu
Qiliang Yuan