Re: BUG: tick device NULL pointer during system initialization andshutdown

[Thomas Gleixner]

On Tue, 25 Jun 2013, Prarit Bhargava wrote:
> On 06/24/2013 09:57 AM, Thomas Gleixner wrote:
> > On Tue, 18 Jun 2013, Prarit Bhargava wrote:
> > 
> >> Similar panics reported during bringup here:
> >>
> >> http://lists.infradead.org/pipermail/linux-arm-kernel/2013-May/166205.html
> >> http://lkml.org/lkml/2013/5/8/342
> >>
> >> I've seen this a few times on 3.10 based kernels.
> >>
> >> [  175.842027] Disabling non-boot CPUs ...
> >> [  475.827017] BUG: unable to handle kernel NULL pointer dereference at
> >> 0000000000000048
> > 
> > That looks like a stale bit in tick_broadcast_force_mask.
> > 
> > Does the patch below fix it?
> >
> 
> Thomas,
> 
> Thanks for the patch.
> 
> The reproducibility appears to be quite low.  I'm seeing this roughly 1 time
> every six hours of continuous system reboots.  I'm testing right now with your
> patch.  I'll update the thread in a couple of days...

I have a proper version of that patch now along with an explanation of
the failure.

-------------------->

Subject: tick: Make oneshot broadcast robust vs. CPU offlining
From: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Date: Wed, 26 Jun 2013 12:17:32 +0200

In periodic mode we remove offline cpus from the broadcast propagation
mask. In oneshot mode we fail to do so. This was not a problem so far,
but the recent changes to the broadcast propagation introduced a
constellation which can result in a NULL pointer dereference.

What happens is:

CPU0			CPU1
			idle()
			  arch_idle()
			    tick_broadcast_oneshot_control(OFF);
			      set cpu1 in tick_broadcast_force_mask
			  if (cpu_offline())
			     arch_cpu_dead()

cpu_dead_cleanup(cpu1)
 cpu1 tickdevice pointer = NULL

broadcast interrupt
  dereference cpu1 tickdevice pointer -> OOPS

We dereference the pointer because cpu1 is still set in
tick_broadcast_force_mask and tick_do_broadcast() expects a valid
cpumask and therefor lacks any further checks.

Remove the cpu from the tick_broadcast_force_mask before we set the
tick device pointer to NULL. Also add a sanity check to the oneshot
broadcast function, so we can detect such issues w/o crashing the
machine.

Reported-by: Prarit Bhargava <prarit@xxxxxxxxxx>
Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
---
 kernel/time/tick-broadcast.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

Index: tip/kernel/time/tick-broadcast.c
===================================================================
--- tip.orig/kernel/time/tick-broadcast.c
+++ tip/kernel/time/tick-broadcast.c
@@ -522,6 +522,13 @@ again:
 	cpumask_clear(tick_broadcast_force_mask);
 
 	/*
+	 * Sanity check. Catch the case where we try to broadcast to
+	 * offline cpus.
+	 */
+	if (WARN_ON_ONCE(!cpumask_subset(tmpmask, cpu_online_mask)))
+		cpumask_and(tmpmask, tmpmask, cpu_online_mask);
+
+	/*
 	 * Wakeup the cpus which have an expired event.
 	 */
 	tick_do_broadcast(tmpmask);
@@ -761,10 +768,12 @@ void tick_shutdown_broadcast_oneshot(uns
 	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
 
 	/*
-	 * Clear the broadcast mask flag for the dead cpu, but do not
-	 * stop the broadcast device!
+	 * Clear the broadcast masks for the dead cpu, but do not stop
+	 * the broadcast device!
 	 */
 	cpumask_clear_cpu(cpu, tick_broadcast_oneshot_mask);
+	cpumask_clear_cpu(cpu, tick_broadcast_pending_mask);
+	cpumask_clear_cpu(cpu, tick_broadcast_force_mask);
 
 	raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
 }
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/