Re: [PATCH 0/2][concept RFC] x86: BIOS-save kernel log to diskupon panic

[Ahmed S. Darwish]

On Tue, Jan 25, 2011 at 03:09:48PM +0100, Ingo Molnar wrote:
> 
> * Ahmed S. Darwish <darwish.07@xxxxxxxxx> wrote:
> 
> > Hi,
> > 
> > I've faced some very early panics in latest kernel. Being a run of the mill
> > x86 laptop, the machine is void of debugging aids like serial ports or
> > network boot.
> > 
> > As a possible solution, below patches prototypes the idea of persistently
> > storing the kernel log ring to a hard disk partition using the enhanced BIOS
> > 0x13 services.

[...]

> > 
> > Diffstat:
> > 
> >  arch/x86/kernel/saveoops-rmode.S |  483 ++++++++++++++++++++++++++++++++++++++
> >  arch/x86/include/asm/saveoops.h  |   15 ++
> >  arch/x86/kernel/saveoops.c       |  219 +++++++++++++++++
> >  arch/x86/kernel/setup.c          |    9 +
> >  arch/x86/kernel/Makefile         |    3 +
> >  lib/Kconfig.debug                |   15 ++
> >  6 files changed, 744 insertions(+), 0 deletions(-)
> 
> Ok, i have to admit that while i'm a rabid BIOS-hater i find this debug feature very 
> very interesting, for the plain reason that if it's implemented in a robust and 
> clever way then this has a chance to improve debuggability of pretty much any Linux 
> laptop quite enormously!
> 
> While we generally thoroughly hate BIOSes from beginning to end, one thing can be 
> said, a BIOS bootstraps very early during bootup, and it's relatively simple to 
> trigger as well.
> 

Yes, the BIOS subset used is the same one used by SYSLINUX, grub, etc to load
the kernel. If the kernel is on, these functions are hopefully quite stable.

The complete __roadblock__ I'm currently facing though is restoring the disk
controllers to the state originally setup by the BIOS Power-on self-test (POST).
I hope such re-initialization is even technically feasible.

Without such re-initialization, we'll just be risking the BIOS code exploding.
That was the case in the 5-minute hang described in the cover sheet (PATCH #0).

> Also, since latest kernels do not stomp on BIOS data structures anymore (low RAM), 
> there's some good chance it's still functional at the point of crash - be that an 
> early crash or a later crash.
> 

Yes, I've noticed that thankfully the kernel reserves the BIOS EBDA area. I'm
not sure though if it reserves the real-mode vector table area 0x0 -> 0x400?

> I think the biggest areas of practical concern would be:
> 
>  - Can this mechanism ever, under any circumstance corrupt any real data, destroy 
>    the MBR or do other nasties. Can you think of any additional fail-safe measures 
>    where you could _further robustify the BIOS calls_ to make sure it can never go 
>    to the wrong sector(s)? I really do not want to think of trusting a BIOS to 
>    _write to my disk_.
> 

Admittedly, I was quite worried while testing this on disk. As a possible
safety trigger, I'm thinking of coding a small script under "tools/" that will
write a cookie in the desired area. If the real-mode code did not find such
cookie in the expected place, it abandons the write operation.

>  - Is there some hidden disk area somewhere on PCs, or somewhere on a real partition
>    on typical Linux distributions, which we could use without having to reinstall
>    the box? This would increase utility and availability enormously. I'm thinking of 
>    partition _ends_ which sometimes get rounded in an awkward way and which are 
>    potentially skipped by most Linux filesystems. Even a very small, 512 bytes of 
>    area would be extremely useful for debugging weird suspend hangs ...
> 

I did not have to re-partition the box here. A kindof a hacky solution was
disabling the swap partition and using it for storing the log. That would make
the feature available without re-installing the box, at the cost of temporarily
disabling swap.

Another testing method was booting the kernel from a USB stick and writing the
log there. If there are other free places in the boot disk, it can be used. But
I'm afraid it will make saveoops much more dangerous than what it already is
(by virtue of being much closer to file-system structures and such).

>  - Could we automate the recovery of the dump, and just put it into the regular 
>    kernel log on the next (successful) bootup (on a feature-enabled kernel)? That 
>    would make the log of the 'previous crash' very conveniently available in dmesg 
>    and the syslog. Tools like kerneloops could make use of it immediately.
> 

If I can make the code reliably work, it can be very very useful in a huge
number of situations such as these and more. The catch is the "If" part :)

> All in one, a very intriguing idea IMO, and the hardest bits (lowlevel x86 
> transition) is all implemented already.
> 

There is a place under arch/x86 that already does the lmode->rmode transition?

That was (so far) the hardest part; I quite searched the tree but did not find
any, so I rolled my own lmode->rmode switch code in the file "saveoops-rmode.S"

> 
> 	Ingo

Thanks,

-- 
Darwish
http://darwish.07.googlepages.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/