Re: [PATCH net-next v2 0/2] kernel: add support to collect hardware logs in crash recovery kernel
From: Eric W. Biederman
Date: Tue Mar 27 2018 - 12:01:01 EST
Rahul Lakkireddy <rahul.lakkireddy@xxxxxxxxxxx> writes:
> On Tuesday, March 03/27/18, 2018 at 18:47:34 +0530, Eric W. Biederman wrote:
>> Rahul Lakkireddy <rahul.lakkireddy@xxxxxxxxxxx> writes:
>>
>> > On Saturday, March 03/24/18, 2018 at 20:50:52 +0530, Eric W. Biederman wrote:
>> >>
>> >> Rahul Lakkireddy <rahul.lakkireddy@xxxxxxxxxxx> writes:
>> >>
>> >> > On production servers running variety of workloads over time, kernel
>> >> > panic can happen sporadically after days or even months. It is
>> >> > important to collect as much debug logs as possible to root cause
>> >> > and fix the problem, that may not be easy to reproduce. Snapshot of
>> >> > underlying hardware/firmware state (like register dump, firmware
>> >> > logs, adapter memory, etc.), at the time of kernel panic will be very
>> >> > helpful while debugging the culprit device driver.
>> >> >
>> >> > This series of patches add new generic framework that enable device
>> >> > drivers to collect device specific snapshot of the hardware/firmware
>> >> > state of the underlying device in the crash recovery kernel. In crash
>> >> > recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/
>> >> > directory, which is copied by user space scripts for post-analysis.
>> >> >
>> >> > A kernel module crashdd is newly added. In crash recovery kernel,
>> >> > crashdd exposes /sys/kernel/crashdd/ directory containing device
>> >> > specific hardware/firmware logs.
>> >>
>> >> Have you looked at instead of adding a sysfs file adding the dumps
>> >> as additional elf notes in /proc/vmcore?
>> >>
>> >
>> > I see the crash recovery kernel's memory is not present in any of the
>> > the PT_LOAD headers. So, makedumpfile is not collecting the dumps
>> > that are in crash recovery kernel's memory.
>> >
>> > Also, are you suggesting exporting the dumps themselves as PT_NOTE
>> > instead? I'll look into doing it this way.
>>
>> Yes. I was suggesting exporting the dumps themselves as PT_NOTE
>> in /proc/vmcore. I think that will allow makedumpfile to collect
>> your new information without modification.
>>
>
> If I export the dumps themselves as PT_NOTE in /proc/vmcore, can the
> crash tool work without modification; i.e can crash tool extract these
> notes?
I believe crash would need to be taught about these notes. This is
something new.
However "readelf -a random_elf_file" does display elf notes, and elf
notes in general are not hard to extract.
What I expect from an enconding in ELF core dump format is a way to
captuer the data, a way to encode the data, and a way to transport the
data to the people who care. Analysis tools are easy enough after the
fact.
Eric