Re: [PATCH v2] Fix lockup related to stop_machine being stuck in__do_softirq.
From: Tejun Heo
Date: Thu Jun 06 2013 - 16:48:29 EST
On Thu, Jun 06, 2013 at 09:10:06AM -0700, greearb@xxxxxxxxxxxxxxx wrote:
> From: Ben Greear <greearb@xxxxxxxxxxxxxxx>
> The stop machine logic can lock up if all but one of
> the migration threads make it through the disable-irq
> step and the one remaining thread gets stuck in
> __do_softirq. The reason __do_softirq can hang is
> that it has a bail-out based on jiffies timeout, but
> in the lockup case, jiffies itself is not incremented.
> To work around this, re-add the max_restart counter in __do_irq
> and stop processing irqs after 10 restarts.
> Thanks to Tejun Heo and Rusty Russell and others for
> helping me track this down.
> This was introduced in 3.9 by commit: c10d73671ad30f5469
> (softirq: reduce latencies).
> It may be worth looking into ath9k to see if it has issues with
> it's irq handler at a later date.
> The hang stack traces look something like this:
Oops, you already posted the second version. :)
> - * We restart softirq processing for at most 2 ms,
> - * and if need_resched() is not set.
> + * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times,
> + * but break the loop if need_resched() is set or after 2 ms.
> * These limits have been established via experimentation.
> * The two things to balance is latency against fairness -
> @@ -204,6 +204,7 @@ EXPORT_SYMBOL(local_bh_enable_ip);
> * should not be able to lock up the box.
> #define MAX_SOFTIRQ_TIME msecs_to_jiffies(2)
> +#define MAX_SOFTIRQ_RESTART 10
As wrote before, a brief explanation on why both are necessary would
be nice. Something like - "the time limit prevents from introducing
excessive latency from softirq handling and the loop limit protects
against softirq runaway which may happen during stop_machine - see
Please cc Linus and also cc stable@xxxxxxxxxxxxxxxx We definitely
want this backported.
Thanks a lot!
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/