Re: [PATCH 2/2] smp/hotplug,lockdep: Annotate cpuhp_state

[Paul E. McKenney]

On Tue, Sep 05, 2017 at 09:52:20AM +0200, Peter Zijlstra wrote:
> After the st->done annotation, lockdep cross-release now complains
> about:
> 
>   CPU0                  CPU1                    CPU2
>   cpuhp_up_callbacks:	takedown_cpu:		cpuhp_thread_fun:
> 
>   cpuhp_state
>                         irq_lock_sparse()
>     irq_lock_sparse()
>                         wait_for_completion()
>                                                 cpuhp_state
>                                                 complete()
> 
> which again spells deadlock, because CPU0 needs to wait for CPU1's
> irq_lock_sparse which will wait for CPU2's completion, which in turn
> waits for CPU0's cpuhp_state.
> 
> Now, this again mixes up and down chains, but now on cpuhp_state.
> 
> Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> Cc: Byungchul Park <max.byungchul.park@xxxxxxxxx>
> Cc: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
> Reported-by: Mike Galbraith <efault@xxxxxx>
> Tested-by: Mike Galbraith <efault@xxxxxx>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@xxxxxxxxxxxxx>
> ---
>  kernel/cpu.c |   12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -67,11 +67,14 @@ struct cpuhp_cpu_state {
>  static DEFINE_PER_CPU(struct cpuhp_cpu_state, cpuhp_state);
>  
>  #if defined(CONFIG_LOCKDEP) && defined(CONFIG_SMP)
> -static struct lock_class_key cpuhp_state_key;
> +static struct lock_class_key cpuhp_state_up_key;
> +#ifdef CONFIG_HOTPLUG_CPU
> +static struct lock_class_key cpuhp_state_down_key;
> +#endif

These two patches work for me if CONFIG_PROVE_LOCKING=y, but
the lockdep_init_map() wants its key argument to exist if
CONFIG_PROVE_LOCKING=n.

Not sure whether it is better to remove the CONFIG_LOCKDEP #if
on the one hand or to make lockdep_init_map() lose the "(void)"
things on the other...

My tests didn't involve failing CPU hotplug operations, so I
didn't run into the issue Thomas was concerned about.

						Thanx, Paul

>  static struct lockdep_map cpuhp_state_lock_map =
> -	STATIC_LOCKDEP_MAP_INIT("cpuhp_state", &cpuhp_state_key);
> +	STATIC_LOCKDEP_MAP_INIT("cpuhp_state-up", &cpuhp_state_up_key);
>  #endif
>  
>  /**
>   * cpuhp_step - Hotplug state machine step
>   * @name:	Name of the step
> @@ -714,6 +718,8 @@ static int __ref _cpu_down(unsigned int
>  	cpus_write_lock();
>  
>  	lockdep_reinit_st_done();
> +	lockdep_init_map(&cpuhp_state_lock_map, "cpuhp_state-down",
> +			 &cpuhp_state_down_key, 0);
>  
>  	cpuhp_tasks_frozen = tasks_frozen;
>  
> @@ -828,6 +834,8 @@ static int _cpu_up(unsigned int cpu, int
>  	cpus_write_lock();
>  
>  	lockdep_reinit_st_done();
> +	lockdep_init_map(&cpuhp_state_lock_map, "cpuhp_state-up",
> +			 &cpuhp_state_up_key, 0);
>  
>  	if (!cpu_present(cpu)) {
>  		ret = -EINVAL;
> 
>