Re: [PATCH v2] kernel/panic/kexec: fix "crash_kexec_post_notifiers" option issue in oops path

[Vivek Goyal]

On Mon, Mar 23, 2015 at 08:19:43AM +0100, Ingo Molnar wrote:
> 
> * Baoquan He <bhe@xxxxxxxxxx> wrote:
> 
> > CC more people ...
> > 
> > On 03/07/15 at 01:31am, "Hatayama, Daisuke/çå åè" wrote:
> > > The commit f06e5153f4ae2e2f3b0300f0e260e40cb7fefd45 introduced
> > > "crash_kexec_post_notifiers" kernel boot option, which toggles
> > > wheather panic() calls crash_kexec() before panic_notifiers and dump
> > > kmsg or after.
> > > 
> > > The problem is that the commit overlooks panic_on_oops kernel boot
> > > option. If it is enabled, crash_kexec() is called directly without
> > > going through panic() in oops path.
> > > 
> > > To fix this issue, this patch adds a check to
> > > "crash_kexec_post_notifiers" in the condition of kexec_should_crash().
> > > 
> > > Also, put a comment in kexec_should_crash() to explain not obvious
> > > things on this patch.
> > > 
> > > Signed-off-by: HATAYAMA Daisuke <d.hatayama@xxxxxxxxxxxxxx>
> > > Acked-by: Baoquan He <bhe@xxxxxxxxxx>
> > > Tested-by: Hidehiro Kawai <hidehiro.kawai.ez@xxxxxxxxxxx>
> > > Reviewed-by: Masami Hiramatsu <masami.hiramatsu.pt@xxxxxxxxxxx>
> > > ---
> > >  include/linux/kernel.h |  3 +++
> > >  kernel/kexec.c         | 11 +++++++++++
> > >  kernel/panic.c         |  2 +-
> > >  3 files changed, 15 insertions(+), 1 deletion(-)
> 
> This is hack upon hack, but why was this crap merged in the first 
> place?
> 
> I see two problems just by cursory review:
> 
> 1)
> 
> Firstly, the real bug in:
> 
>   f06e5153f4ae ("kernel/panic.c: add "crash_kexec_post_notifiers" option for kdump after panic_notifers")
> 
> Was that crash_kexec() was called unconditionally after notifiers were 
> called, which should be fixed via the simple patch below (untested). 
> Looks much simpler than your fix.
> 

Hi Ingo,

Agreed. Your patch looks good.

> 2)
> 
> Secondly, and more importantly, the whole premise of commit 
> f06e5153f4ae is broken IMHO:
> 
>  "This can help rare situations where kdump fails because of unstable
>   crashed kernel or hardware failure (memory corruption on critical
>   data/code)"
> 
> wtf?
> 
> If the kernel crashed due to a kernel crash, then the kernel booting 
> up in whatever hardware state should be able to do a clean bootup. The 
> fix for those 'rare situations' should be to fix the real bug (for 
> example by making hardware driver init (or deinit) sequences more 
> robust), not to paper it over by ordering around crash-time sequences 
> ...
> 
> If it crashed due to some hardware failure, there's literally an 
> infinite amount of failure modes that may or may not be impacted by 
> kexec crash-time handling ordering. We don't want to put a zillion 
> such flags into the kernel proper just to allow the perturbation of 
> the kernel.

I think one of the motivations behind this patch was call to kmsg_dump().
Some vendors have been wanting to have the capability to save kernel logs
to some NVRAM before transition to second kernel happens. Their argument
is that kdump does not succeed all the time and if kdump does not succeed
then atleast they have something to work with (kernel logs retrieved
from pstore interface).

Not that I agree fully with this as problem might happen while we try
to run panic_notifiers or kmsg_dump hooks and never transition into
kdump kernel.

And it has been literally years since some developers have been pushing for
allowing to run panic notifiers before crash_kexec(). Eric Biederman has been
pushing back saying it reduces the reliability of kdump operation so this
is not acceptable.

So while it is very hacky, this command line option was intorduced which
allowed to override default crash_kexec() behavior and those who want
to do additional things (at their own risk) before transition to second
kernel, can specify this parameter.

Thanks
Vivek

> 
> diff --git a/kernel/panic.c b/kernel/panic.c
> index 8136ad76e5fd..774614f72cbd 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -142,7 +142,8 @@ void panic(const char *fmt, ...)
>  	 * Note: since some panic_notifiers can make crashed kernel
>  	 * more unstable, it can increase risks of the kdump failure too.
>  	 */
> -	crash_kexec(NULL);
> +	if (crash_kexec_post_notifiers)
> +		crash_kexec(NULL);
>  
>  	bust_spinlocks(0);
>  
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/