Re: [PATCH 2/2] smp/hotplug,lockdep: Annotate cpuhp_state

From: Paul E. McKenney
Date: Tue Sep 19 2017 - 13:57:40 EST


On Tue, Sep 05, 2017 at 09:52:20AM +0200, Peter Zijlstra wrote:
> After the st->done annotation, lockdep cross-release now complains
> about:
>
> CPU0 CPU1 CPU2
> cpuhp_up_callbacks: takedown_cpu: cpuhp_thread_fun:
>
> cpuhp_state
> irq_lock_sparse()
> irq_lock_sparse()
> wait_for_completion()
> cpuhp_state
> complete()
>
> which again spells deadlock, because CPU0 needs to wait for CPU1's
> irq_lock_sparse which will wait for CPU2's completion, which in turn
> waits for CPU0's cpuhp_state.
>
> Now, this again mixes up and down chains, but now on cpuhp_state.
>
> Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> Cc: Byungchul Park <max.byungchul.park@xxxxxxxxx>
> Cc: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
> Reported-by: Mike Galbraith <efault@xxxxxx>
> Tested-by: Mike Galbraith <efault@xxxxxx>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@xxxxxxxxxxxxx>
> ---
> kernel/cpu.c | 12 ++++++++++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -67,11 +67,14 @@ struct cpuhp_cpu_state {
> static DEFINE_PER_CPU(struct cpuhp_cpu_state, cpuhp_state);
>
> #if defined(CONFIG_LOCKDEP) && defined(CONFIG_SMP)
> -static struct lock_class_key cpuhp_state_key;
> +static struct lock_class_key cpuhp_state_up_key;
> +#ifdef CONFIG_HOTPLUG_CPU
> +static struct lock_class_key cpuhp_state_down_key;
> +#endif

These two patches work for me if CONFIG_PROVE_LOCKING=y, but
the lockdep_init_map() wants its key argument to exist if
CONFIG_PROVE_LOCKING=n.

Not sure whether it is better to remove the CONFIG_LOCKDEP #if
on the one hand or to make lockdep_init_map() lose the "(void)"
things on the other...

My tests didn't involve failing CPU hotplug operations, so I
didn't run into the issue Thomas was concerned about.

Thanx, Paul

> static struct lockdep_map cpuhp_state_lock_map =
> - STATIC_LOCKDEP_MAP_INIT("cpuhp_state", &cpuhp_state_key);
> + STATIC_LOCKDEP_MAP_INIT("cpuhp_state-up", &cpuhp_state_up_key);
> #endif
>
> /**
> * cpuhp_step - Hotplug state machine step
> * @name: Name of the step
> @@ -714,6 +718,8 @@ static int __ref _cpu_down(unsigned int
> cpus_write_lock();
>
> lockdep_reinit_st_done();
> + lockdep_init_map(&cpuhp_state_lock_map, "cpuhp_state-down",
> + &cpuhp_state_down_key, 0);
>
> cpuhp_tasks_frozen = tasks_frozen;
>
> @@ -828,6 +834,8 @@ static int _cpu_up(unsigned int cpu, int
> cpus_write_lock();
>
> lockdep_reinit_st_done();
> + lockdep_init_map(&cpuhp_state_lock_map, "cpuhp_state-up",
> + &cpuhp_state_up_key, 0);
>
> if (!cpu_present(cpu)) {
> ret = -EINVAL;
>
>