Re: [RFC PATCH v3 0/7] device backed vmemmap crash dump support

From: Zhijian Li (Fujitsu)
Date: Thu Mar 21 2024 - 02:58:24 EST

Next message: Michael S. Tsirkin: "Re: [PATCH v3] vhost/vdpa: Add MSI translation tables to iommu for software-managed MSI"
Previous message: Ayush Tiwari: "[PATCH v3] staging: rtl8712: Fix line length exceeding 100 columns"
In reply to: Baoquan He: "Re: [RFC PATCH v3 0/7] device backed vmemmap crash dump support"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Baoquan,

On 21/03/2024 14:17, Baoquan He wrote:
> On 03/21/24 at 05:40am, Zhijian Li (Fujitsu) wrote:
>> ping
>>
>>
>> Any comment is welcome.
>
> I will have a look at this from kdump side. How do you test your code?

Thanks for your support.

- nothing change is required for makedumpfile and crash-utils
- nothing change is required for kexe-utils if you are using kexec_file_load(2)
- kexec-tool needs apply below patch if you are using kexec_load(2),

After the coredump is collected, crash-utils is able to inspect the device backed vmemmap memory.

From 4a2f0f91cdd8b084bb4ebafdc4f71b8e2a333720 Mon Sep 17 00:00:00 2001
From: Li Zhijian <lizhijian@xxxxxxxxxxx>
Date: Fri, 1 Mar 2024 13:44:36 +0800
Subject: [PATCH] consider device_backed_vmemmap dumpable

Signed-off-by: Li Zhijian <lizhijian@xxxxxxxxxxx>
---
kexec/arch/i386/crashdump-x86.c | 2 ++
kexec/arch/i386/kexec-x86-common.c | 3 +++
kexec/crashdump-elf.c | 2 +-
kexec/kexec.h | 1 +
4 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index a01031e570aa..757922dc3a57 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -284,6 +284,8 @@ static int get_crash_memory_ranges(struct memory_range **range, int *ranges,
type = RANGE_PRAM;
} else if(memcmp(str,"Persistent Memory\n",18) == 0 ) {
type = RANGE_PMEM;
+ } else if (memcmp(str, "Device Backed Vmemmap\n", 22)) {
+ type = RANGE_DEVICE_BACKED_VMEMMAP;
} else if(memcmp(str,"reserved\n",9) == 0 ) {
type = RANGE_RESERVED;
} else if (memcmp(str, "Reserved\n", 9) == 0) {
diff --git a/kexec/arch/i386/kexec-x86-common.c b/kexec/arch/i386/kexec-x86-common.c
index ffc95a9e43f8..0dd76903e7fc 100644
--- a/kexec/arch/i386/kexec-x86-common.c
+++ b/kexec/arch/i386/kexec-x86-common.c
@@ -111,6 +111,9 @@ static int get_memory_ranges_proc_iomem(struct memory_range **range, int *ranges
else if (memcmp(str, "Persistent Memory\n", 18) == 0) {
type = RANGE_PMEM;
}
+ else if (memcmp(str, "Device Backed Vmemmap\n", 22)) {
+ type = RANGE_DEVICE_BACKED_VMEMMAP;
+ }
else {
continue;
}
diff --git a/kexec/crashdump-elf.c b/kexec/crashdump-elf.c
index b8bb686a17ca..98ad854e3f8d 100644
--- a/kexec/crashdump-elf.c
+++ b/kexec/crashdump-elf.c
@@ -199,7 +199,7 @@ int FUNC(struct kexec_info *info,
* A seprate program header for Backup Region*/
for (i = 0; i < ranges; i++, range++) {
unsigned long long mstart, mend;
- if (range->type != RANGE_RAM)
+ if (range->type != RANGE_RAM || range->type != RANGE_DEVICE_BACKED_VMEMMAP)
continue;
mstart = range->start;
mend = range->end;
diff --git a/kexec/kexec.h b/kexec/kexec.h
index 1004aff15945..c0481bb2727d 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -139,6 +139,7 @@ struct memory_range {
#define RANGE_UNCACHED 4
#define RANGE_PMEM 6
#define RANGE_PRAM 11
+#define RANGE_DEVICE_BACKED_VMEMMAP 12
};

struct memory_ranges {
--
2.29.2

>
> By the way, there's issue reported by test robot.

I see, i have fixed it locally which doesn't matter for this feature(it fails in !ZONE_DEVICE)

Thanks
Zhijian

>
> Thanks
> Baoquan
>
>>
>>
>> On 06/03/2024 18:28, Li Zhijian wrote:
>>> Hello folks,
>>>
>>> Compared with the V2[1] I posted a long time ago, this time it is a
>>> completely new proposal design.
>>>
>>> ### Background and motivate overview ###
>>> ---
>>> Crash dump is an important feature for troubleshooting the kernel. It is the
>>> final way to chase what happened at the kernel panic, slowdown, and so on. It
>>> is one of the most important tools for customer support.
>>>
>>> Currently, there are 2 syscalls(kexec_file_load(2) and kexec_load(2)) to
>>> configure the dumpable regions. Generally, (A)iomem resources registered with
>>> flags (IORESOURCE_SYSTEM_RAM | IORESOUCE_BUSY) for kexec_file_load(2) or
>>> (B)iomem resources registered with "System RAM" name prefix for kexec_load(2)
>>> are dumpable.
>>>
>>> The pmem use cases including fsdax and devdax, could map their vmemmap to
>>> their own devices. In this case, these part of vmemmap will not be dumped when
>>> crash happened since these regions are satisfied with neither the above (A)
>>> nor (B).
>>>
>>> In fsdax, the vmemmap(struct page array) becomes very important, it is one of
>>> the key data to find status of reverse map. Lacking of the information may
>>> cause difficulty to analyze trouble around pmem (especially Filesystem-DAX).
>>> That means troubleshooters are unable to check more details about pmem from
>>> the dumpfile.
>>>
>>> ### Proposal ###
>>> ---
>>> In this proposal, register the device backed vmemmap as a separate resource.
>>> This resource has its own new flag and name, and then teaches kexec_file_load(2)
>>> and kexec_load(2) to mark it as dumpable.
>>>
>>> Proposed flag: IORESOURCE_DEVICE_BACKED_VMEMMAP
>>> Proposed name: "Device Backed Vmemmap"
>>>
>>> NOTE: crash-utils also needs to adapt to this new name for kexec_load()
>>>
>>> With current proposal, the /proc/iomem should show as following for device
>>> backed vmemmap
>>> # cat /proc/iomem
>>> ...
>>> fffc0000-ffffffff : Reserved
>>> 100000000-13fffffff : Persistent Memory
>>> 100000000-10fffffff : namespace0.0
>>> 100000000-1005fffff : Device Backed Vmemmap # fsdax
>>> a80000000-b7fffffff : CXL Window 0
>>> a80000000-affffffff : Persistent Memory
>>> a80000000-affffffff : region1
>>> a80000000-a811fffff : namespace1.0
>>> a80000000-a811fffff : Device Backed Vmemmap # devdax
>>> a81200000-abfffffff : dax1.0
>>> b80000000-c7fffffff : CXL Window 1
>>> c80000000-147fffffff : PCI Bus 0000:00
>>> c80000000-c801fffff : PCI Bus 0000:01
>>> ...
>>>
>>> ### Kdump service reloading ###
>>> ---
>>> Once the kdump service is loaded, if changes to CPUs or memory occur,
>>> either by hot un/plug or off/onlining, the crash elfcorehdr should also
>>> be updated. There are 2 approaches to make the reloading more efficient.
>>> 1) Use udev rules to watch CPU and memory events, then reload kdump
>>> 2) Enable kernel crash hotplug to automatically reload elfcorehdr (>= 6.5)
>>>
>>> This reloading also needed when device backed vmemmap layouts change, Similar
>>> to what 1) does now, one could add the following as the first lines to the
>>> RHEL udev rule file /usr/lib/udev/rules.d/98-kexec.rules:
>>>
>>> # namespace updated: watch daxX.Y(devdax) and pfnX.Y(fsdax) of nd
>>> SUBSYSTEM=="nd", KERNEL=="[dp][af][xn][0-9].*", ACTION=="bind", GOTO="kdump_reload"
>>> SUBSYSTEM=="nd", KERNEL=="[dp][af][xn][0-9].*", ACTION=="unbind", GOTO="kdump_reload"
>>> # devdax <-> system-ram updated: watch daxX.Y of dax
>>> SUBSYSTEM=="dax", KERNEL=="dax[0-9].*", ACTION=="bind", GOTO="kdump_reload"
>>> SUBSYSTEM=="dax", KERNEL=="dax[0-9].*", ACTION=="unbind", GOTO="kdump_reload"
>>>
>>> Regarding 2), my idea is that it would need to call the memory_notify() in
>>> devm_memremap_pages_release() and devm_memremap_pages() to trigger the crash
>>> hotplug. This part is not yet mature, but it does not affect the whole feature
>>> because we can still use method 1) alternatively.
>>>
>>> [1] https://lore.kernel.org/lkml/02066f0f-dbc0-0388-4233-8e24b6f8435b@xxxxxxxxxxx/T/
>>> --------------------------------------------
>>> changes from V2[1]
>>> - new proposal design
>>>
>>> CC: Alison Schofield <alison.schofield@xxxxxxxxx>
>>> CC: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>>> CC: Baoquan He <bhe@xxxxxxxxxx>
>>> CC: Borislav Petkov <bp@xxxxxxxxx>
>>> CC: Dan Williams <dan.j.williams@xxxxxxxxx>
>>> CC: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
>>> CC: Dave Jiang <dave.jiang@xxxxxxxxx>
>>> CC: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
>>> CC: "H. Peter Anvin" <hpa@xxxxxxxxx>
>>> CC: Ingo Molnar <mingo@xxxxxxxxxx>
>>> CC: Ira Weiny <ira.weiny@xxxxxxxxx>
>>> CC: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
>>> CC: Vishal Verma <vishal.l.verma@xxxxxxxxx>
>>> CC: linux-cxl@xxxxxxxxxxxxxxx
>>> CC: linux-mm@xxxxxxxxx
>>> CC: nvdimm@xxxxxxxxxxxxxxx
>>> CC: x86@xxxxxxxxxx
>>>
>>> Li Zhijian (7):
>>> mm: memremap: register/unregister altmap region to a separate resource
>>> mm: memremap: add pgmap_parent_resource() helper
>>> nvdimm: pmem: assign a parent resource for vmemmap region for the
>>> fsdax
>>> dax: pmem: assign a parent resource for vmemmap region for the devdax
>>> resource: Introduce walk device_backed_vmemmap res() helper
>>> x86/crash: make device backed vmemmap dumpable for kexec_file_load
>>> nvdimm: set force_raw=1 in kdump kernel
>>>
>>> arch/x86/kernel/crash.c | 5 +++++
>>> drivers/dax/pmem.c | 8 ++++++--
>>> drivers/nvdimm/namespace_devs.c | 3 +++
>>> drivers/nvdimm/pmem.c | 9 ++++++---
>>> include/linux/ioport.h | 4 ++++
>>> include/linux/memremap.h | 4 ++++
>>> kernel/resource.c | 13 +++++++++++++
>>> mm/memremap.c | 30 +++++++++++++++++++++++++++++-
>>> 8 files changed, 70 insertions(+), 6 deletions(-)
>>>
>

Next message: Michael S. Tsirkin: "Re: [PATCH v3] vhost/vdpa: Add MSI translation tables to iommu for software-managed MSI"
Previous message: Ayush Tiwari: "[PATCH v3] staging: rtl8712: Fix line length exceeding 100 columns"
In reply to: Baoquan He: "Re: [RFC PATCH v3 0/7] device backed vmemmap crash dump support"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]