Re: NOHZ: WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule,round 2

[Frederic Weisbecker]

2013/5/20 Borislav Petkov <bp@xxxxxxxxx>:
> On Mon, May 20, 2013 at 11:16:33AM +0800, Michael Wang wrote:
>> I suppose the reason is that the cpu we passed to
>> mod_delayed_work_on() has a chance to become offline before we
>> disabled irq, what about check it before send resched ipi? like:
>
> I think this is only addressing the symptoms - what we should be doing
> instead is asking ourselves why are we even scheduling work on a cpu if
> the machine goes offline?
>
> I don't know though who should be responsible for killing all that
> work - the workqueue itself or the guy who created it, i.e. cpufreq
> governor...
>
> Hmmm.

Let's look at this portion of cpu_down():

	err = __stop_machine(take_cpu_down, &tcd_param, cpumask_of(cpu));
	if (err) {
		/* CPU didn't die: tell everyone.  Can't complain. */
		smpboot_unpark_threads(cpu);
		cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
		goto out_release;
	}
	BUG_ON(cpu_online(cpu));

	/*
	 * The migration_call() CPU_DYING callback will have removed all
	 * runnable tasks from the cpu, there's only the idle task left now
	 * that the migration thread is done doing the stop_machine thing.
	 *
	 * Wait for the stop thread to go away.
	 */
	while (!idle_cpu(cpu))
		cpu_relax();
	/* This actually kills the CPU. */
	__cpu_die(cpu);

	/* CPU is completely dead: tell everyone.  Too late to complain. */
	cpu_notify_nofail(CPU_DEAD | mod, hcpu);

	check_for_tasks(cpu);

The CPU is considered offline after the take_cpu_down stop machine job
completes. But the struct timer_list timers are migrated later through
CPU_DEAD notification. Only once that's completed we check for illegal
residual tasks in the CPU.  So there is a little window between the
stop machine thing and __cpu_die() where a timer can fire with
cpu_online(cpu) == 1.

Now concerning the workqueue I don't know. I guess the per cpu ones
are not migrated due to their affinity. Apparently they can still wake
up and execute works due to the timers...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/