Re: uninitialized pmem struct pages

[Michal Hocko]

On Tue 05-01-21 10:13:49, David Hildenbrand wrote:
> On 05.01.21 10:05, Michal Hocko wrote:
> > On Tue 05-01-21 00:57:43, Dan Williams wrote:
> >> On Tue, Jan 5, 2021 at 12:42 AM Michal Hocko <mhocko@xxxxxxxx> wrote:
> >>>
> >>> On Tue 05-01-21 00:27:34, Dan Williams wrote:
> >>>> On Tue, Jan 5, 2021 at 12:17 AM Michal Hocko <mhocko@xxxxxxxx> wrote:
> >>>>>
> >>>>> On Tue 05-01-21 09:01:00, Michal Hocko wrote:
> >>>>>> On Mon 04-01-21 16:44:52, David Hildenbrand wrote:
> >>>>>>> On 04.01.21 16:43, David Hildenbrand wrote:
> >>>>>>>> On 04.01.21 16:33, Michal Hocko wrote:
> >>>>>>>>> On Mon 04-01-21 16:15:23, David Hildenbrand wrote:
> >>>>>>>>>> On 04.01.21 16:10, Michal Hocko wrote:
> >>>>>>>>> [...]
> >>>>>>>>>> Do the physical addresses you see fall into the same section as boot
> >>>>>>>>>> memory? Or what's around these addresses?
> >>>>>>>>>
> >>>>>>>>> Yes I am getting a garbage for the first struct page belonging to the
> >>>>>>>>> pmem section [1]
> >>>>>>>>> [    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
> >>>>>>>>> [    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile
> >>>>>>>>>
> >>>>>>>>> The pfn without the initialized struct page is 0x6060000. This is a
> >>>>>>>>> first pfn in a section.
> >>>>>>>>
> >>>>>>>> Okay, so we're not dealing with the "early section" mess I described,
> >>>>>>>> different story.
> >>>>>>>>
> >>>>>>>> Due to [1], is_mem_section_removable() called
> >>>>>>>> pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not
> >>>>>>>> initialized.
> >>>>>>>>
> >>>>>>>> Let's assume this is indeed a reserved pfn in the altmap. What's the
> >>>>>>>> actual address of the memmap?
> >>>>>>>>
> >>>>>>>> I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually
> >>>>>>>> part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?
> >>>>>>>>
> >>>>>>>> If it's not self-hosted, initializing the relevant memmaps should work
> >>>>>>>> just fine I guess. Otherwise things get more complicated.
> >>>>>>>
> >>>>>>> Oh, I forgot: pfn_to_online_page() should at least in your example make
> >>>>>>> sure other pfn walkers are safe. It was just an issue of
> >>>>>>> is_mem_section_removable().
> >>>>>>
> >>>>>> Hmm, I suspect you are right. I haven't put this together, thanks! The memory
> >>>>>> section is indeed marked offline so pfn_to_online_page would indeed bail
> >>>>>> out:
> >>>>>> crash> p (0x6060000>>15)
> >>>>>> $3 = 3084
> >>>>>> crash> p mem_section[3084/128][3084 & 127]
> >>>>>> $4 = {
> >>>>>>   section_mem_map = 18446736128020054019,
> >>>>>>   usage = 0xffff902dcf956680,
> >>>>>>   page_ext = 0x0,
> >>>>>>   pad = 0
> >>>>>> }
> >>>>>> crash> p 18446736128020054019 & (1UL<<2)
> >>>>>> $5 = 0
> >>>>>>
> >>>>>> That makes it considerably less of a problem than I thought!
> >>>>>
> >>>>> Forgot to add that those who are running kernels without 53cdc1cb29e8
> >>>>> ("drivers/base/memory.c: indicate all memory blocks as removable") for
> >>>>> some reason can fix the crash by the following simple patch.
> >>>>>
> >>>>> Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
> >>>>> ===================================================================
> >>>>> --- linux-5.3-users_mhocko_SLE15-SP2_for-next.orig/drivers/base/memory.c
> >>>>> +++ linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
> >>>>> @@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
> >>>>>                 goto out;
> >>>>>
> >>>>>         for (i = 0; i < sections_per_block; i++) {
> >>>>> -               if (!present_section_nr(mem->start_section_nr + i))
> >>>>> +               unsigned long nr = mem->start_section_nr + i;
> >>>>> +               if (!present_section_nr(nr))
> >>>>>                         continue;
> >>>>> -               pfn = section_nr_to_pfn(mem->start_section_nr + i);
> >>>>> +               if (!online_section_nr()) {
> >>>>
> >>>> I assume that's onlince_section_nr(nr) in the version that compiles?
> >>>
> >>> Yup.
> >>>
> >>>> This makes sense because the memory block size is larger than the
> >>>> section size. I suspect you have 1GB memory block size on this system,
> >>>> but since the System RAM and PMEM collide at a 512MB alignment in a
> >>>> memory block you end up walking the back end of the last 512MB of the
> >>>> System RAM memory block and run into the offline PMEM section.
> >>>
> >>> Sections are 128MB and memory blocks are 2GB on this system.
> >>>
> >>>> So, I don't think it's pfn_to_online_page that necessarily needs to
> >>>> know how to disambiguate each page, it's things that walk sections and
> >>>> memory blocks and expects them to be consistent over the span.
> >>>
> >>> Well, memory hotplug code is hard wired to sparse memory model so in
> >>> this particular case asking about the section is ok. But pfn walkers
> >>> shouldn't really care and only rely on pfn_to_online_page. But that will
> >>> do the right thing here. So we are good as long as the section is marked
> >>> properly. But this would become a problem as soon as the uninitialized
> >>> pages where sharing the same memory section as David pointed out.
> >>> pfn_to_online_page would then return something containing garbage. So we
> >>> should still think of a way to either initialize all those pages or make
> >>> sure pfn_to_online_page recognizes them. The former is preferred IMHO.
> >>
> >> The former would not have saved the crash in this case because
> >> pfn_to_online_page() is not used in v5.3:removable_show() that I can
> >> see, nor some of the other paths that might walk pfns and the wrong
> >> thing with ZONE_DEVICE.
> > 
> > If the page was initialized properly, and by that I mean also have it
> > reserved, then the old code would have properly reported is as not
> > removable.
> > 
> >> However, I do think pfn_to_online_page() should be reliable, and I
> >> prefer to just brute force add a section flag to indicate whether the
> >> section might be ZONE_DEVICE polluted and fallback to the
> >> get_dev_pagemap() slow-path in that case.
> > 
> > Do we have some spare room to hold that flag in a section?
> > 
> >> ...but it would still require hunting to find the places where
> >> pfn_to_online_page() is missing for assumptions like this crash which
> >> assumed memblock-online + section-present == section-online.
> > 
> > Yes, but most users should be using pfn_to_online_page already.
> > 
> 
> Quite honestly, let's not hack around this issue and just fix it
> properly - make pfn_to_online_page() only ever return an initialized,
> online (buddy) page, just as documented.

Just to make sure we are on the same page. You are agreeing with Dan
that pfn_to_online_page should check for zone device pages? Ideally in a
slow path.

> What speaks against not adding early sections in case they have a
> relevant hole at the end, besides wasting some MB? Relevant setups most
> probably just don't care. Print a warning.

If we can avoid shared section reasonably then I do not object.
Sacrificing few Megs of memory should be fine in most cases. It is not
like reasonable systems would have many hybrid sections.

-- 
Michal Hocko
SUSE Labs