Re: list corruption in deferred_split_scan()

[Qian Cai]

On Thu, 2019-07-18 at 17:59 -0700, Yang Shi wrote:
> 
> On 7/18/19 5:54 PM, Qian Cai wrote:
> > 
> > > On Jul 12, 2019, at 3:12 PM, Yang Shi <yang.shi@xxxxxxxxxxxxxxxxx> wrote:
> > > 
> > > 
> > > 
> > > On 7/11/19 2:07 PM, Qian Cai wrote:
> > > > On Wed, 2019-07-10 at 17:16 -0700, Yang Shi wrote:
> > > > > Hi Qian,
> > > > > 
> > > > > 
> > > > > Thanks for reporting the issue. But, I can't reproduce it on my
> > > > > machine.
> > > > > Could you please share more details about your test? How often did you
> > > > > run into this problem?
> > > > 
> > > > I can almost reproduce it every time on a HPE ProLiant DL385 Gen10
> > > > server. Here
> > > > is some more information.
> > > > 
> > > > # cat .config
> > > > 
> > > > https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
> > > 
> > > I tried your kernel config, but I still can't reproduce it. My compiler
> > > doesn't have retpoline support, so CONFIG_RETPOLINE is disabled in my
> > > test, but I don't think this would make any difference for this case.
> > > 
> > > According to the bug call trace in the earlier email, it looks deferred
> > > _split_scan lost race with put_compound_page. The put_compound_page would
> > > call free_transhuge_page() which delete the page from the deferred split
> > > queue, but it may still appear on the deferred list due to some reason.
> > > 
> > > Would you please try the below patch?
> > > 
> > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> > > index b7f709d..66bd9db 100644
> > > --- a/mm/huge_memory.c
> > > +++ b/mm/huge_memory.c
> > > @@ -2765,7 +2765,7 @@ int split_huge_page_to_list(struct page *page,
> > > struct list_head *list)
> > > ÂÂÂÂÂÂÂÂÂif (!mapcount && page_ref_freeze(head, 1 + extra_pins)) {
> > > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif (!list_empty(page_deferred_list(head))) {
> > > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂds_queue->split_queue_len--;
> > > -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlist_del(page_deferred_list(head));
> > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlist_del_init(page_deferred_list(head));
> > > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ}
> > > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂif (mapping)
> > > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ__dec_node_page_state(page, NR_SHMEM_THPS);
> > > @@ -2814,7 +2814,7 @@ void free_transhuge_page(struct page *page)
> > > ÂÂÂÂÂÂÂÂÂspin_lock_irqsave(&ds_queue->split_queue_lock, flags);
> > > ÂÂÂÂÂÂÂÂÂif (!list_empty(page_deferred_list(page))) {
> > > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂds_queue->split_queue_len--;
> > > -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlist_del(page_deferred_list(page));
> > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlist_del_init(page_deferred_list(page));
> > > ÂÂÂÂÂÂÂÂÂ}
> > > ÂÂÂÂÂÂÂÂÂspin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);
> > > ÂÂÂÂÂÂÂÂÂfree_compound_page(page);
> > 
> > Unfortunately, I am no longer be able to reproduce the original list
> > corruption with todayâs linux-next.
> 
> It is because the patches have been dropped from -mm tree by Andrew dueÂ
> to this problem I guess. You have to use next-20190711, or apply theÂ
> patches on today's linux-next.
> 

The patch you have here does not help. Only applied the part for
free_transhuge_page() as you requested.

[ÂÂ375.006307][ T3580] list_del corruption. next->prev should be
ffffea0030e10098, but was ffff888ea8d0cdb8
[ÂÂ375.015928][ T3580] ------------[ cut here ]------------
[ÂÂ375.021296][ T3580] kernel BUG at lib/list_debug.c:56!
[ÂÂ375.026491][ T3580] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN NOPTI
[ÂÂ375.033680][ T3580] CPU: 84 PID: 3580 Comm: oom01 Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.2.0-next-20190711+ #2
[ÂÂ375.042964][ T3580] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 06/24/2019
[ÂÂ375.052256][ T3580] RIP: 0010:__list_del_entry_valid+0xa8/0xb6
[ÂÂ375.058135][ T3580] Code: de 48 c7 c7 c0 5a b3 b0 e8 b9 fa bc ff 0f 0b 48 c7
c7 60 a0 21 b1 e8 13 52 01 00 4c 89 e6 48 c7 c7 20 5b b3 b0 e8 9c fa bc ff <0f>
0b 48 c7 c7 20 a0 21 b1 e8 f6 51 01 00 4c 89 ea 48 89 de 48 c7
[ÂÂ375.077722][ T3580] RSP: 0018:ffff888ebc4b73c0 EFLAGS: 00010082
[ÂÂ375.083684][ T3580] RAX: 0000000000000054 RBX: ffffea0030e10098 RCX:
ffffffffb015d728
[ÂÂ375.091566][ T3580] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
ffff88903263d380
[ÂÂ375.099448][ T3580] RBP: ffff888ebc4b73d8 R08: ffffed12064c7a71 R09:
ffffed12064c7a70
[ÂÂ375.107330][ T3580] R10: ffffed12064c7a70 R11: ffff88903263d387 R12:
ffffea0030e10098
[ÂÂ375.115212][ T3580] R13: ffffea0031d40098 R14: ffffea0030e10034 R15:
ffffea0031d40098
[ÂÂ375.123095][ T3580] FS:ÂÂ00007fc3dc851700(0000) GS:ffff889032600000(0000)
knlGS:0000000000000000
[ÂÂ375.131937][ T3580] CS:ÂÂ0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ÂÂ375.138421][ T3580] CR2: 00007fc25fa39000 CR3: 0000000884762000 CR4:
00000000001406a0
[ÂÂ375.146301][ T3580] Call Trace:
[ÂÂ375.149472][ T3580]ÂÂdeferred_split_scan+0x337/0x740
[ÂÂ375.154475][ T3580]ÂÂ? split_huge_page_to_list+0xe30/0xe30
[ÂÂ375.160002][ T3580]ÂÂ? __sched_text_start+0x8/0x8
[ÂÂ375.164743][ T3580]ÂÂ? __radix_tree_lookup+0x12d/0x1e0
[ÂÂ375.169923][ T3580]ÂÂdo_shrink_slab+0x244/0x5a0
[ÂÂ375.174490][ T3580]ÂÂshrink_slab+0x253/0x440
[ÂÂ375.178794][ T3580]ÂÂ? unregister_shrinker+0x110/0x110
[ÂÂ375.183972][ T3580]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ375.188715][ T3580]ÂÂ? mem_cgroup_protected+0x20f/0x260
[ÂÂ375.193976][ T3580]ÂÂ? shrink_node+0x1ad/0xa30
[ÂÂ375.198453][ T3580]ÂÂshrink_node+0x31e/0xa30
[ÂÂ375.202755][ T3580]ÂÂ? shrink_node_memcg+0x1560/0x1560
[ÂÂ375.207934][ T3580]ÂÂ? ktime_get+0x93/0x110
[ÂÂ375.212147][ T3580]ÂÂdo_try_to_free_pages+0x22f/0x820
[ÂÂ375.217236][ T3580]ÂÂ? shrink_node+0xa30/0xa30
[ÂÂ375.221711][ T3580]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ375.226450][ T3580]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ375.231277][ T3580]ÂÂtry_to_free_pages+0x242/0x4d0
[ÂÂ375.236102][ T3580]ÂÂ? do_try_to_free_pages+0x820/0x820
[ÂÂ375.241370][ T3580]ÂÂ__alloc_pages_nodemask+0x9ce/0x1bc0
[ÂÂ375.246721][ T3580]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ375.251459][ T3580]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ375.256722][ T3580]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ375.261458][ T3580]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ375.266287][ T3580]ÂÂ? do_anonymous_page+0x343/0xe30
[ÂÂ375.271289][ T3580]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ375.276029][ T3580]ÂÂ? __count_memcg_events+0x8b/0x1c0
[ÂÂ375.281204][ T3580]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ375.285945][ T3580]ÂÂ? __lru_cache_add+0x122/0x160
[ÂÂ375.290774][ T3580]ÂÂalloc_pages_vma+0x89/0x2c0
[ÂÂ375.295339][ T3580]ÂÂdo_anonymous_page+0x3e1/0xe30
[ÂÂ375.300168][ T3580]ÂÂ? __update_load_avg_cfs_rq+0x2c/0x490
[ÂÂ375.305692][ T3580]ÂÂ? finish_fault+0x120/0x120
[ÂÂ375.310257][ T3580]ÂÂ? alloc_pages_vma+0x21e/0x2c0
[ÂÂ375.315085][ T3580]ÂÂhandle_pte_fault+0x457/0x12c0
[ÂÂ375.319912][ T3580]ÂÂ__handle_mm_fault+0x79a/0xa50
[ÂÂ375.324738][ T3580]ÂÂ? vmf_insert_mixed_mkwrite+0x20/0x20
[ÂÂ375.330175][ T3580]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ375.334913][ T3580]ÂÂ? __count_memcg_events+0x8b/0x1c0
[ÂÂ375.340090][ T3580]ÂÂhandle_mm_fault+0x17f/0x370
[ÂÂ375.344745][ T3580]ÂÂ__do_page_fault+0x25b/0x5d0
[ÂÂ375.349398][ T3580]ÂÂdo_page_fault+0x4c/0x2cf
[ÂÂ375.353793][ T3580]ÂÂ? page_fault+0x5/0x20
[ÂÂ375.357920][ T3580]ÂÂpage_fault+0x1b/0x20
[ÂÂ375.361959][ T3580] RIP: 0033:0x410be0
[ÂÂ375.365737][ T3580] Code: 89 de e8 e3 23 ff ff 48 83 f8 ff 0f 84 86 00 00 00
48 89 c5 41 83 fc 02 74 28 41 83 fc 03 74 62 e8 95 29 ff ff 31 d2 48 98 90 <c6>
44 15 00 07 48 01 c2 48 39 d3 7f f3 31 c0 5b 5d 41 5c c3 0f 1f
[ÂÂ375.385323][ T3580] RSP: 002b:00007fc3dc850ec0 EFLAGS: 00010206
[ÂÂ375.391283][ T3580] RAX: 0000000000001000 RBX: 00000000c0000000 RCX:
00007fda6c168497
[ÂÂ375.399164][ T3580] RDX: 00000000041e9000 RSI: 00000000c0000000 RDI:
0000000000000000
[ÂÂ375.407047][ T3580] RBP: 00007fc25b850000 R08: 00000000ffffffff R09:
0000000000000000
[ÂÂ375.414928][ T3580] R10: 0000000000000022 R11: 0000000000000246 R12:
0000000000000001
[ÂÂ375.422812][ T3580] R13: 00007ffc4a58701f R14: 0000000000000000 R15:
00007fc3dc850fc0
[ÂÂ375.430694][ T3580] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat
kvm_amd kvm ses enclosure irqbypass dax_pmem dax_pmem_core efivars ip_tables
x_tables xfs sd_mod smartpqi scsi_transport_sas mlx5_core tg3 firmware_class
libphy dm_mirror dm_region_hash dm_log dm_mod efivarfs
[ÂÂ375.455820][ T3580] ---[ end trace 82d52f9627313e53 ]---
[ÂÂ375.461172][ T3580] RIP: 0010:__list_del_entry_valid+0xa8/0xb6
[ÂÂ375.467048][ T3580] Code: de 48 c7 c7 c0 5a b3 b0 e8 b9 fa bc ff 0f 0b 48 c7
c7 60 a0 21 b1 e8 13 52 01 00 4c 89 e6 48 c7 c7 20 5b b3 b0 e8 9c fa bc ff <0f>
0b 48 c7 c7 20 a0 21 b1 e8 f6 51 01 00 4c 89 ea 48 89 de 48 c7
[ÂÂ375.486635][ T3580] RSP: 0018:ffff888ebc4b73c0 EFLAGS: 00010082
[ÂÂ375.492597][ T3580] RAX: 0000000000000054 RBX: ffffea0030e10098 RCX:
ffffffffb015d728
[ÂÂ375.500479][ T3580] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
ffff88903263d380
[ÂÂ375.508361][ T3580] RBP: ffff888ebc4b73d8 R08: ffffed12064c7a71 R09:
ffffed12064c7a70
[ÂÂ375.516244][ T3580] R10: ffffed12064c7a70 R11: ffff88903263d387 R12:
ffffea0030e10098
[ÂÂ375.524124][ T3580] R13: ffffea0031d40098 R14: ffffea0030e10034 R15:
ffffea0031d40098
[ÂÂ375.532007][ T3580] FS:ÂÂ00007fc3dc851700(0000) GS:ffff889032600000(0000)
knlGS:0000000000000000
[ÂÂ375.540851][ T3580] CS:ÂÂ0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ÂÂ375.547335][ T3580] CR2: 00007fc25fa39000 CR3: 0000000884762000 CR4:
00000000001406a0
[ÂÂ375.555217][ T3580] Kernel panic - not syncing: Fatal exception
[ÂÂ376.868640][ T3580] Shutting down cpus with NMI
[ÂÂ376.873223][ T3580] Kernel Offset: 0x2ec00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ÂÂ376.884878][ T3580] ---[ end Kernel panic - not syncing: Fatal exception ]---