Re: [PATCH -tip] fix race between stop_two_cpus and stop_cpus

[Prarit Bhargava]

On 11/01/2013 07:08 AM, Mel Gorman wrote:
> On Thu, Oct 31, 2013 at 04:31:44PM -0400, Rik van Riel wrote:
>> There is a race between stop_two_cpus, and the global stop_cpus.
>>
> 
> What was the trigger for this? I want to see what was missing from my own
> testing. I'm going to go out on a limb and guess that CPU hotplug was also
> running in the background to specifically stress this sort of rare condition.
> Something like running a standard test with the monitors/watch-cpuoffline.sh
> from mmtests running in parallel.
> 

I have a test that loads and unloads each module in /lib/modules/3.*/...

Each run typically takes a few minutes.  After running 4-5 times, the system
issues a soft lockup warning with a CPU in multi_cpu_stop().  Unfortunately,
kdump isn't working on this particular system (due to another bug) so I modified
the code with (sorry for the cut-and-paste):

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 05039e3..4a8c9f9 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -323,8 +323,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtime
                else
                        dump_stack();

-               if (softlockup_panic)
+               if (softlockup_panic) {
+                       show_state();
                        panic("softlockup: hung tasks");
+               }
                __this_cpu_write(soft_watchdog_warn, true);
        } else
                __this_cpu_write(soft_watchdog_warn, false);

and then 'echo 1 > /proc/sys/kernel/softlockup_panic' to get a full trace of all
tasks.

When I did this and ran the kernel module load unload test ...

[prarit@prarit tmp]$ cat /tmp/intel.log | grep RIP
[  678.081168] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.156180] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.230190] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.244186] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.259194] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.274192] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.288195] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.303197] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.318200] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.333203] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.349206] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.364208] RIP: 0010:[<ffffffff810d328b>]  [<ffffffff810d328b>]
multi_cpu_stop+0x7b/0xf0
[  678.379211] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.394212] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.409215] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.424217] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.438219] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.452221] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.466228] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.481228] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.496230] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.511234] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.526236] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.541238] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.556244] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.571243] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.586247] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0
[  678.601248] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.616251] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  678.632254] RIP: 0010:[<ffffffff810d328b>]  [<ffffffff810d328b>]
multi_cpu_stop+0x7b/0xf0
[  678.647257] RIP: 0010:[<ffffffff810d3292>]  [<ffffffff810d3292>]
multi_cpu_stop+0x82/0xf0
[  687.570464] RIP: 0010:[<ffffffff810d3296>]  [<ffffffff810d3296>]
multi_cpu_stop+0x86/0xf0

and,

[prarit@prarit tmp]$ cat /tmp/intel.log | grep RIP | wc -l
32

which shows all 32 cpus are "correctly" in the cpu stop threads.  After some
investigation, Rik came up with his patch.

Hope this explains things,

P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/