Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

From: Feng Tang
Date: Mon Oct 22 2018 - 05:49:12 EST


Hi maintainers,

On Thu, Oct 11, 2018 at 05:59:41PM +0800, Feng Tang wrote:
> Hi Peter,
>
> On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> > On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > > Sometimes when debugging kernel panic, we saw many extra noisy error
> > > messages after the expected end:
> > >
> > > [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > > [ 35.749975] ------------[ cut here ]------------
> > >
> > > These messages may overflow the sceen (framebuffer) and make debugging
> > > much difficulter.
> >
> > *blink* you actually using the framebuffer for debugging ?! Why the heck
> > are you doing that?
>
> I didn't make it clear.
>
> I just wanted to show some use case, that when I work under command line
> mode, if the kernel panic (without reboot configured), it will pop out
> the panic message, and these noisy message may keep poping up and I
> hardly see the actual panic message on the screen.
>
> >
> > > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > > index c93fcfd..b703862 100644
> > > --- a/arch/x86/kernel/process.c
> > > +++ b/arch/x86/kernel/process.c
> > > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> > > * Remove this CPU:
> > > */
> > > set_cpu_online(smp_processor_id(), false);
> > > + set_cpu_active(smp_processor_id(), false);
> > > disable_local_APIC();
> > > mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> > >
> >
> > WTH is stop_this_cpu() and how do we even get here with active still
> > set?
>
> Current panic() will call
> smp_send_stop
> native_stop_other_cpus
> apic->send_IPI_allbutself(REBOOT_VECTOR)
> smp_reboot_interrupt
> stop_this_cpu
>
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 7fc4a37..cf41b7b 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> > > {
> > > int ilb = cpumask_first(nohz.idle_cpus_mask);
> > >
> > > - if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > > + if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> > > return ilb;
> > >
> > > return nr_cpu_ids;
> >
> >
> > Similar, this is the result of taking the CPU away without going through
> > the normal path. You're doing something dodgy.
>
> You are right, it's hacky. As I said in the commit log, suggestions are
> appreciated.
>
> Current emergercy handling of panic didn't notify the idle subsystem of
> other cpus being stopped.
>
> I have tried several other ways:
> a) add a panic notifier to cancel the schdule timer when panic happens
> b) don't re-enable the local interrupt inside panic() after normal panic
> handling.

Could you help to review the patch? thanks,

- Feng