Re: [PATCH RT 10/30] hrtimer: Prevent using hrtimer_grab_expiry_lock() on migration_base

[Rasmus Villemoes]

On 23/01/2020 21.39, Steven Rostedt wrote:
> 4.19.94-rt39-rc2 stable review patch.
> If anyone has any objections, please let me know.
> 
> ------------------
> 
> From: Julien Grall <julien.grall@xxxxxxx>
> 
> [ Upstream commit cef1b87f98823af923a386f3f69149acb212d4a1 ]
> 
> As tglx puts it:
> |If base == migration_base then there is no point to lock soft_expiry_lock
> |simply because the timer is not executing the callback in soft irq context
> |and the whole lock/unlock dance can be avoided.
> 
> Furthermore, all the path leading to hrtimer_grab_expiry_lock() assumes
> timer->base and timer->base->cpu_base are always non-NULL. So it is safe
> to remove the NULL checks here.
> 
> Signed-off-by: Julien Grall <julien.grall@xxxxxxx>
> Link: https://lkml.kernel.org/r/alpine.DEB.2.21.1908211557420.2223@xxxxxxxxxxxxxxxxxxxxxxx
> Signed-off-by: Steven Rostedt (VMware) <rostedt@xxxxxxxxxxx>
> [bigeasy: rewrite changelog]
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>

Pretty late to the party, but I think I've bisected a problem to this
patch (and its required fixup for !SMP, "hrtimer: Add a missing bracket
and hide `migration_base on !SMP").

Originally, a customer reported that upgrading from 4.19.82-rt30 to
v4.19.106-rt45 failed to boot, stalling around the time the network gets
initialized (this is a board with an embedded Marvell switch). Perhaps 1
in 10 times, the board would come up successfully. I haven't been able
to reproduce that particular problem (or, perhaps I've seen it once or
twice, but not nearly often enough to use that as a basis for bisection).

However, building with their rescue initrd and booting that, the board
would consistently hang during reboot. Sometimes I would get lines like

[   72.956630] sched: RT throttling activated
[   72.973769] lanx: port 1(lan1) entered disabled state
[   73.000401] lanx: port 2(lan2) entered disabled state
[   73.974951] lanx: port 3(lan3) entered disabled state
[   73.997473] lanx: port 4(lan4) entered disabled state
[   74.968006] lanx: port 5(lan5) entered disabled state

other times there would be no output, but the board was still hanging.
Reverting

b1a471ec4df1 - hrtimer: Prevent using hrtimer_grab_expiry_lock() on
migration_base
40aae5708e7a - hrtimer: Add a missing bracket and hide `migration_base'
on !SMP

on top of v4.19.94-rt39 makes that problem go away, i.e. the board
reboots as expected.

The board is a 32 bit powerpc (mpc8309) !SMP. Any ideas what I can do to
debug this further?

Thanks,
Rasmus