Re: [PATCH 2/4] workqueue: Improve scalability of workqueue watchdog touch
From: Nicholas Piggin
Date: Tue Jun 25 2024 - 20:53:01 EST
On Wed Jun 26, 2024 at 2:57 AM AEST, Tejun Heo wrote:
> On Tue, Jun 25, 2024 at 09:42:45PM +1000, Nicholas Piggin wrote:
> > On a ~2000 CPU powerpc system, hard lockups have been observed in the
> > workqueue code when stop_machine runs (in this case due to CPU hotplug).
> > This is due to lots of CPUs spinning in multi_cpu_stop, calling
> > touch_nmi_watchdog() which ends up calling wq_watchdog_touch().
> > wq_watchdog_touch() writes to the global variable wq_watchdog_touched,
> > and that can find itself in the same cacheline as other important
> > workqueue data, which slows down operations to the point of lockups.
> >
> > In the case of the following abridged trace, worker_pool_idr was in
> > the hot line, causing the lockups to always appear at idr_find.
> >
> > watchdog: CPU 1125 self-detected hard LOCKUP @ idr_find
> > Call Trace:
> > get_work_pool
> > __queue_work
> > call_timer_fn
> > run_timer_softirq
> > __do_softirq
> > do_softirq_own_stack
> > irq_exit
> > timer_interrupt
> > decrementer_common_virt
> > * interrupt: 900 (timer) at multi_cpu_stop
> > multi_cpu_stop
> > cpu_stopper_thread
> > smpboot_thread_fn
> > kthread
> >
> > Fix this by having wq_watchdog_touch() only write to the line if the
> > last time a touch was recorded exceeds 1/4 of the watchdog threshold.
> >
> > Reported-by: Srikar Dronamraju <srikar@xxxxxxxxxxxxxxxxxx>
> > Signed-off-by: Nicholas Piggin <npiggin@xxxxxxxxx>
>
> Applied 1-2 to wq/for-6.11.
Thanks Tejun.
> I think 3 and 4 should probably be routed
> through either tip or Andrew?
Yeah, let's see if it gets any comments.
Thanks,
Nick