Re: [tip:smp/hotplug] cpu/hotplug: Plug death reporting race

From: Paul E. McKenney
Date: Thu Mar 03 2016 - 09:11:28 EST


On Thu, Mar 03, 2016 at 02:03:35AM -0800, tip-bot for Thomas Gleixner wrote:
> Commit-ID: 71f87b2fc64c2e9b6d53cb817f28711b959d3dfe
> Gitweb: http://git.kernel.org/tip/71f87b2fc64c2e9b6d53cb817f28711b959d3dfe
> Author: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> AuthorDate: Thu, 3 Mar 2016 10:52:10 +0100
> Committer: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> CommitDate: Thu, 3 Mar 2016 10:52:10 +0100
>
> cpu/hotplug: Plug death reporting race
>
> Paul noticed that the conversion of the death reporting introduced a race
> where the outgoing cpu might be delayed after waking the controll processor,
> so it might not be able to call rcu_report_dead() before being physically
> removed, leading to RCU stalls.
>
> We cant call complete after rcu_report_dead(), so instead of going back to
> busy polling, simply issue a function call to do the completion.
>
> Fixes: 27d50c7eeb0f "rcu: Make CPU_DYING_IDLE an explicit call"
> Reported-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx>
> Link: http://lkml.kernel.org/r/20160302201127.GA23440@xxxxxxxxxxxxxxxxxx
> Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> Acked-by: Peter Zijlstra <peterz@xxxxxxxxxxxxx>

There are some possible issues with printk()s from WARN_ON() calls
using RCU from within smp_call_function_single(), but in normal
execution this should be OK. Besides, the WARN_ON() contains a
cpu_online(this_cpu), so it simply won't fire in this case.

So nice fix!!!

Reviewed-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx>

> ---
> kernel/cpu.c | 16 ++++++++++++++--
> 1 file changed, 14 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index ff8059b..93e9d89 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -755,14 +755,26 @@ static int notify_dead(unsigned int cpu)
> return 0;
> }
>
> +static void cpuhp_complete_idle_dead(void *arg)
> +{
> + struct cpuhp_cpu_state *st = arg;
> +
> + complete(&st->done);
> +}
> +
> void cpuhp_report_idle_dead(void)
> {
> struct cpuhp_cpu_state *st = this_cpu_ptr(&cpuhp_state);
>
> BUG_ON(st->state != CPUHP_AP_OFFLINE);
> - st->state = CPUHP_AP_IDLE_DEAD;
> - complete(&st->done);
> rcu_report_dead(smp_processor_id());
> + st->state = CPUHP_AP_IDLE_DEAD;
> + /*
> + * We cannot call complete after rcu_report_dead() so we delegate it
> + * to an online cpu.
> + */
> + smp_call_function_single(cpumask_first(cpu_online_mask),
> + cpuhp_complete_idle_dead, st, 0);
> }
>
> #else
>