Re: [PATCHv2] arm64:kexec: have own crash_smp_send_stop() for crash dump for nonpanic cores

[Hoeun Ryu]

Hello, James.

Thank you for the meticulous test and review.

On Fri, 2017-08-11 at 18:02 +0100, James Morse wrote:
> Hi Hoeun,
> 
> On 07/08/17 06:09, Hoeun Ryu wrote:
> > 
> > ÂCommit 0ee5941 : (x86/panic: replace smp_send_stop() with kdump friendly
> > version in panic path) introduced crash_smp_send_stop() which is a weak
> > function and can be overriden by architecture codes to fix the side effect
> (overridden)

It'll be fixed in the next version.

> 
> 
> > 
> > caused by commit f06e515 : (kernel/panic.c: add "crash_kexec_post_
> > notifiers" option).
> > 
> > ÂARM64 architecture uses the weak version function and the problem is that
> > the weak function simply calls smp_send_stop() which makes other CPUs
> > offline and takes away the chance to save crash information for nonpanic
> > CPUs in machine_crash_shutdown() when crash_kexec_post_notifiers kernel
> > option is enabled.
> > 
> > ÂCalling smp_send_crash_stop() in machine_crash_shutdown() is useless
> > because all nonpanic CPUs are already offline by smp_send_stop() in this
> > case and smp_send_crash_stop() only works against online CPUs.
> 
> > 
> > ÂThe result is that /proc/vmcore is not available with the error messages;
> > "Warning: Zero PT_NOTE entries found", "Kdump: vmcore not initialized".
> When I tried this I got one of these warnings for each secondary CPU, but the
> vmcore file was still available. When I ran 'crash' on the vmcore it reported:
> > 
> > CPUS: 6 [OFFLINE: 5]
> Did I miss as step to reproduce this? If not, can we change this paragraph to
> say something like:
> > 
> > The result is that secondary CPUs registers are not saved by crash_save_cpu()
> > and the vmcore file misreports these CPUs as being offline.

Actually the commit log comes from the patch to fix a similar issue in arm port.
I'll change the commit log with yours.

> 
> > 
> > Âcrash_smp_send_stop() is implemented to fix this problem by replacing the
> > exising smp_send_crash_stop() and adding a check for multiple calling to
> (existing)

It'll be fixed in the next version.

> 
> 
> > 
> > the function. The function (strong symbol version) saves crash information
> > for nonpanic CPUs and machine_crash_shutdown() tries to save crash
> > information for nonpanic CPUs only when crash_kexec_post_notifiers kernel
> > option is disabled.
> > 
> > * crash_kexec_post_notifiers : false
> > 
> > Â panic()
> > ÂÂÂÂ__crash_kexec()
> > ÂÂÂÂÂÂmachine_crash_shutdown()
> > ÂÂÂÂÂÂÂÂcrash_smp_send_stop()ÂÂÂÂ<= save crash dump for nonpanic cores
> > 
> > * crash_kexec_post_notifiers : true
> > 
> > Â panic()
> > ÂÂÂÂcrash_smp_send_stop()ÂÂÂÂÂÂÂÂ<= save crash dump for nonpanic cores
> > ÂÂÂÂ__crash_kexec()
> > ÂÂÂÂÂÂmachine_crash_shutdown()
> > ÂÂÂÂÂÂÂÂcrash_smp_send_stop()ÂÂÂÂ<= just return.
> 
> > 
> > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> > index dc66e6e..73d8f5e 100644
> > --- a/arch/arm64/kernel/smp.c
> > +++ b/arch/arm64/kernel/smp.c
> > @@ -977,11 +977,21 @@ void smp_send_stop(void)
> > Â}
> > Â
> > Â#ifdef CONFIG_KEXEC_CORE
> > -void smp_send_crash_stop(void)
> > +void crash_smp_send_stop(void)
> > Â{
> > +	static int cpus_stopped;
> > Â	cpumask_t mask;
> > Â	unsigned long timeout;
> > Â
> > +	/*
> > +	Â* This function can be called twice in panic path, but obviously
> > +	Â* we execute this only once.
> > +	Â*/
> > +	if (cpus_stopped)
> > +		return;
> > +
> > +	cpus_stopped = 1;
> > +
> This cpus_stopped=1 can't happen on multiple CPUs at the same time as any second
> call is guaranteed to be on the same CPU, both are behind panic()s
> 'atomic_cmpxchg()'.

'cpu_stopped' variable is not for the race of multi CPUs.
This variable is simply to prevent from calling
'smp_cross_call(&mask, IPI_CPU_CRASH_STOP)' twice in the machine_crash_shutdown().
Please look at following call path.
Â
* crash_kexec_post_notifiers : true
Âpanic()
  Âcrash_smp_send_stop() {
    Â...
    Âcpu_stopped = 1       Â<= make it '1'
    Âsmp_cross_call()       <= save crash dump for nonpanic cores
  Â}
ÂÂ Â __crash_kexec()
    Âmachine_crash_shutdown()
      Âcrash_smp_send_stop() {Â
        Âif (cpu_stopped)
          Âreturn      <= just return.
      Â}
> 
> 
> Other than my '/proc/vmcore is not available' question above, this looks fine to me:
> Reviewed-by: James Morse <james.morse@xxxxxxx>
> Tested-by: James Morse <james.morse@xxxxxxx>
> 
> 
> Thanks!
> 
> James
> 
> 
>