Re: [PATCH, RFC] reimplement flush_workqueue()

[Oleg Nesterov]

On 01/04, Srivatsa Vaddagiri wrote:
>
> On Tue, Dec 19, 2006 at 03:43:19AM +0300, Oleg Nesterov wrote:
> > > Taking workqueue_mutex() unconditionally in flush_workqueue() means
> > > that we'll deadlock if a single-threaded workqueue callback handler calls
> > > flush_workqueue().
> > 
> > Well. But flush_workqueue() drops workqueue_mutex before going to sleep ?
> 
> ... and acquires it again after woken from sleep. That can be a problem, which 
> will lead to the problem described here:
> 
> 	http://lkml.org/lkml/2006/12/7/374
> 
> In brief:
> 
> keventd thread					hotplug thread
> --------------					--------------
> 
>   run_workqueue()
> 	|
>      work_fn()
> 	 |
> 	flush_workqueue()
> 	     |	
> 	   flush_cpu_workqueue
> 		|				cpu_down()
> 	     mutex_unlock(wq_mutex);		     |
> 	(above opens window for hotplug)	   mutex_lock(wq_mutex);
>     		|				   /* bring down cpu */	
> 	     wait_for_completition();		     notifier(CPU_DEAD, ..)
> 		| 				       workqueue_cpu_callback
> 		| 				        cleanup_workqueue_thread
> 		|					  kthread_stop()
> 		|
> 		|
> 	     mutex_lock(wq_mutex); <- Can deadlock
> 
> 
> The kthread_stop() will wait for keventd() thread to exit, but keventd()
> is blocked on mutex_lock(wq_mutex) leading to a deadlock.

Thanks, I need to think about this.

However I am not sure I fully understand the problem.

First, this deadlock was not introduced by recent changes (including "single
threaded flush_workqueue() takes workqueue_mutex too"), yes?

Also, it seems to me we have a much more simple scenario for deadlock.

events/0 runs run_workqueue(), work->func() sleeps or takes a preemtion. CPU 0
dies, keventd thread migrates to another CPU. CPU_DEAD calls kthread_stop() under
workqueue_mutex and waits for until kevents thread exits. Now, if this work (or
another work pending on cwq->worklist) takes workqueue_mutex (for example, does
flush_workqueue) we have a deadlock.

No?

Oleg.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/