Re: [PATCH v1] drivers/base/memory.c: indicate all memory blocks as removable

[David Hildenbrand]

> Am 27.03.2020 um 23:13 schrieb Dan Williams <dan.j.williams@xxxxxxxxx>:
> 
> ïOn Fri, Mar 27, 2020 at 9:50 AM David Hildenbrand <david@xxxxxxxxxx> wrote:
>> 
>>> On 27.03.20 17:28, Dan Williams wrote:
>>> On Fri, Mar 27, 2020 at 2:00 AM David Hildenbrand <david@xxxxxxxxxx> wrote:
>>>> 
>>>> On 27.03.20 08:47, Michal Hocko wrote:
>>>>> On Thu 26-03-20 23:24:08, Dan Williams wrote:
>>>>> [...]
>>>>>> David, Andrew,
>>>>>> 
>>>>>> I'd like to recommend this patch for -stable as it likely (test
>>>>>> underway) solves this crash report from Steve:
>>>>>> 
>>>>>> [  148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p))
>>>>>> [  148.796074] ------------[ cut here ]------------
>>>>>> [  148.796098] kernel BUG at include/linux/mm.h:1087!
>>>>>> [  148.796126] invalid opcode: 0000 [#1] SMP NOPTI
>>>>>> [  148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8=
>>>>>> 6_64+debug #1
>>>>>> [  148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5=
>>>>>> C620.86B.02.01.0010.010620200716 01/06/2020
>>>>>> [  148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0
>>>>>> [  148.796561] Call Trace:
>>>>>> [  148.796591]  removable_show+0x6e/0xa0
>>>>>> [  148.796608]  dev_attr_show+0x19/0x40
>>>>>> [  148.796625]  sysfs_kf_seq_show+0xa9/0x100
>>>>>> [  148.796640]  seq_read+0xd5/0x450
>>>>>> [  148.796657]  vfs_read+0xc5/0x180
>>>>>> [  148.796672]  ksys_read+0x68/0xe0
>>>>>> [  148.796688]  do_syscall_64+0x5c/0xa0
>>>>>> [  148.796704]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
>>>>>> [  148.796721] RIP: 0033:0x7f3ab1646412
>>>>>> 
>>>>>> ...on a non-debug kernel it just crashes.
>>>>>> 
>>>>>> In this case lsmem is failing when reading memory96:
>>>>>> 
>>>>>> openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC) = 4
>>>>>> fcntl(4, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
>>>>>> fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
>>>>>> read(4,  <unfinished ...>)              = ?
>>>>>> +++ killed by SIGSEGV +++
>>>>>> Segmentation fault (core dumped)
>>>>>> 
>>>>>> ...which is phys_index 0x60 => memory address 0x3000000000
>>>>>> 
>>>>>> On this platform that lands us here:
>>>>>> 
>>>>>> 100000000-303fffffff : System RAM
>>>>>>  291f000000-291fe00f70 : Kernel code
>>>>>>  2920000000-292051efff : Kernel rodata
>>>>>>  2920600000-292093b0bf : Kernel data
>>>>>>  29214f3000-2922dfffff : Kernel bss
>>>>>> 3040000000-305fffffff : Reserved
>>>>>> 3060000000-1aa5fffffff : Persistent Memory
>>>>> 
>>>>> OK, 2GB memblocks and that would mean [0x3000000000, 0x3080000000]
>>>>> 
>>>>>> ...where the last memory block of System RAM is shared with persistent
>>>>>> memory. I.e. the block is only partially online which means that
>>>>>> page_to_nid() in is_mem_section_removable() will assert or crash for
>>>>>> some of the offline pages in that block.
>>>>> 
>>>>> Yes, this patch is a simple workaround. Normal memory hotplug will not
>>>>> blow up because it should be able to find out that test_pages_in_a_zone
>>>>> is false. Who knows how other potential pfn walkers handle that.
>>>> 
>>>> All other pfn walkers now correctly use pfn_to_online_page() - which
>>>> will also result in false positives in this scenario and is still to be
>>>> fixed by Dan IIRC. [1]
>>> 
>>> Sorry, it's been too long and this fell out of my cache. I also turned
>>> away once the major fire in KVM was put out with special consideration
>>> for for devmem pages. What's left these days? ...besides
>>> removable_show()?
>> 
>> Essentially any pfn_to_online_page() is a candidate.
>> 
>> E.g.,
>> 
>> mm/memory-failure.c:memory_failure()
>> 
>> is obviously broken (could be worked around)
> 
> Ooh, the current state looks worse than when I looked previously. I
> wasn't copied on commit 96c804a6ae8c ("mm/memory-failure.c: don't
> access uninitialized memmaps in memory_failure()"). That commit seems
> to ensure the pmem errors in memory sections that overlap with
> System-RAM are not handled. So that change looks broken to me.
> Previously get_devpagemap() was sufficient protection.
> 

Well, it went in before we learned that pfn_to_online_page() is now broken in corner cases since sub-section hotadd.


>> 
>> Also
>> 
>> mm/memory-failure.c:soft_offline_page()
>> 
>> is obviously broken.
> 
> How exactly? The soft_offline_page() callers seem to already account
> for System-RAM vs devmem.

Then my quick scan was maybe wrong :)

> 
>> 
>> 
>> Also set_zone_contiguous()->__pageblock_pfn_to_page() is broken, when it
>> checks for "page_zone(start_page) != zone" if the memmap contains garbage.
>> 
>> And I only checked a handful of examples.
> 
> Ok, but as the first example shows in the absence of a problem report
> these pre-emptive changes might make things worse so I don't think
> it's as simple as go instrument all the pfn_to_online_page() users.
> 

Fixing pfn_to_online_page() is the right thing to do, not working around it eventually having false positives IMHO.