Re: list corruption in deferred_split_scan()
From: Qian Cai
Date: Wed Jul 24 2019 - 17:13:54 EST
On Wed, 2019-07-10 at 17:43 -0400, Qian Cai wrote:
> Running LTP oom01 test case with swap triggers a crash below. Revert the
> series
> "Make deferred split shrinker memcg aware" [1] seems fix the issue.
You might want to look harder on this commit, as reverted it alone on the top of
5.2.0-next-20190711 fixed the issue.
aefde94195ca mm: thp: make deferred split shrinker memcg aware [1]
[1] https://lore.kernel.org/linux-mm/1561507361-59349-5-git-send-email-yang.shi@
linux.alibaba.com/
There are all console output while running LTP oom01 before the crash that might
be useful.
[ÂÂ656.302886][ T3384] WARNING: CPU: 79 PID: 3384 at mm/page_alloc.c:4608
__alloc_pages_nodemask+0x1a8a/0x1bc0
[ÂÂ656.304395][ T3409] kmemleak: Cannot allocate a kmemleak_object structure
[ÂÂ656.312714][ T3384] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat
kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core efivars ip_tables
x_tables xfs sd_mod smartpqi scsi_transport_sas mlx5_core tg3 libphy
firmware_class dm_mirror dm_region_hash dm_log dm_mod efivarfs
[ÂÂ656.320916][ T3409] kmemleak: Kernel memory leak detector disabled
[ÂÂ656.344509][ T3384] CPU: 79 PID: 3384 Comm: oom01 Not tainted 5.2.0-next-
20190711+ #3
[ÂÂ656.344523][ T3384] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 06/24/2019
[ÂÂ656.352100][ÂÂT829] kmemleak: Automatic memory scanning thread ended
[ÂÂ656.358648][ T3384] RIP: 0010:__alloc_pages_nodemask+0x1a8a/0x1bc0
[ÂÂ656.358658][ T3384] Code: 00 85 d2 0f 85 a1 00 00 00 48 c7 c7 e0 29 c3 a3 e8
3b 98 62 00 65 48 8b 1c 25 80 ee 01 00 e9 85 fa ff ff 0f 0b e9 3e fb ff ff <0f>
0b 48 8b b5 00 ff ff ff 8b 8d 84 fe ff ff 48 c7 c2 00 1d 6c a3
[ÂÂ656.358675][ T3384] RSP: 0000:ffff888efa4a6210 EFLAGS: 00010046
[ÂÂ656.406140][ T3384] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
ffffffffa2b28be2
[ÂÂ656.414033][ T3384] RDX: 0000000000000000 RSI: dffffc0000000000 RDI:
ffffffffa4d15d60
[ÂÂ656.421926][ T3384] RBP: ffff888efa4a6420 R08: fffffbfff49a2bad R09:
fffffbfff49a2bac
[ÂÂ656.429818][ T3384] R10: fffffbfff49a2bac R11: 0000000000000003 R12:
ffffffffa4d15d60
[ÂÂ656.437711][ T3384] R13: 0000000000000000 R14: 0000000000000800 R15:
0000000000000000
[ÂÂ656.445605][ T3384] FS:ÂÂ00007ff44adfc700(0000) GS:ffff889032f80000(0000)
knlGS:0000000000000000
[ÂÂ656.454459][ T3384] CS:ÂÂ0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ÂÂ656.460952][ T3384] CR2: 00007ff2f05e1000 CR3: 0000001012e44000 CR4:
00000000001406a0
[ÂÂ656.468843][ T3384] Call Trace:
[ÂÂ656.472026][ T3384]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ656.477303][ T3384]ÂÂ? stack_depot_save+0x215/0x58b
[ÂÂ656.482228][ T3384]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ656.486976][ T3384]ÂÂ? stack_depot_save+0x183/0x58b
[ÂÂ656.491900][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.496647][ T3384]ÂÂ? do_raw_spin_unlock+0xa8/0x140
[ÂÂ656.501658][ T3384]ÂÂ? stack_depot_save+0x215/0x58b
[ÂÂ656.506582][ T3384]ÂÂalloc_pages_current+0x9c/0x110
[ÂÂ656.511505][ T3384]ÂÂallocate_slab+0x351/0x11f0
[ÂÂ656.516077][ T3384]ÂÂ? kasan_slab_alloc+0x11/0x20
[ÂÂ656.520824][ T3384]ÂÂnew_slab+0x46/0x70
[ÂÂ656.524702][ T3384]ÂÂ? pageout.isra.4+0x3e5/0xa00
[ÂÂ656.529449][ T3384]ÂÂ___slab_alloc+0x5d4/0x9c0
[ÂÂ656.533933][ T3384]ÂÂ? try_to_free_pages+0x242/0x4d0
[ÂÂ656.538941][ T3384]ÂÂ? __alloc_pages_nodemask+0x9ce/0x1bc0
[ÂÂ656.544476][ T3384]ÂÂ? alloc_pages_vma+0x89/0x2c0
[ÂÂ656.549226][ T3384]ÂÂ? __do_page_fault+0x25b/0x5d0
[ÂÂ656.554064][ T3384]ÂÂ? create_object+0x3a/0x3e0
[ÂÂ656.558637][ T3384]ÂÂ? init_object+0x7e/0x90
[ÂÂ656.562947][ T3384]ÂÂ? create_object+0x3a/0x3e0
[ÂÂ656.567520][ T3384]ÂÂ__slab_alloc+0x12/0x20
[ÂÂ656.571742][ T3384]ÂÂ? __slab_alloc+0x12/0x20
[ÂÂ656.576142][ T3384]ÂÂkmem_cache_alloc+0x32a/0x400
[ÂÂ656.580890][ T3384]ÂÂcreate_object+0x3a/0x3e0
[ÂÂ656.585291][ T3384]ÂÂ? stack_depot_save+0x183/0x58b
[ÂÂ656.590215][ T3384]ÂÂkmemleak_alloc+0x71/0xa0
[ÂÂ656.594611][ T3384]ÂÂkmem_cache_alloc+0x272/0x400
[ÂÂ656.599361][ T3384]ÂÂ? ___might_sleep+0xab/0xc0
[ÂÂ656.603934][ T3384]ÂÂ? mempool_free+0x170/0x170
[ÂÂ656.608507][ T3384]ÂÂmempool_alloc_slab+0x2d/0x40
[ÂÂ656.613254][ T3384]ÂÂmempool_alloc+0x10a/0x29e
[ÂÂ656.617739][ T3384]ÂÂ? alloc_pages_vma+0x89/0x2c0
[ÂÂ656.622485][ T3384]ÂÂ? mempool_resize+0x390/0x390
[ÂÂ656.627233][ T3384]ÂÂ? __read_once_size_nocheck.constprop.2+0x10/0x10
[ÂÂ656.633730][ T3384]ÂÂbio_alloc_bioset+0x150/0x330
[ÂÂ656.638477][ T3384]ÂÂ? bvec_alloc+0x1b0/0x1b0
[ÂÂ656.642892][ T3384]ÂÂalloc_io+0x2f/0x230 [dm_mod]
[ÂÂ656.647654][ T3384]ÂÂ__split_and_process_bio+0x99/0x630 [dm_mod]
[ÂÂ656.653714][ T3384]ÂÂ? blk_rq_map_sg+0x9f0/0x9f0
[ÂÂ656.658388][ T3384]ÂÂ? __send_empty_flush.constprop.11+0x1f0/0x1f0 [dm_mod]
[ÂÂ656.665407][ T3384]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ656.670244][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.674992][ T3384]ÂÂ? blk_queue_split+0x60/0x90
[ÂÂ656.679654][ T3384]ÂÂ? __blk_queue_split+0x970/0x970
[ÂÂ656.684679][ T3384]ÂÂdm_process_bio+0x33f/0x520 [dm_mod]
[ÂÂ656.690054][ T3384]ÂÂ? __process_bio+0x230/0x230 [dm_mod]
[ÂÂ656.695515][ T3384]ÂÂdm_make_request+0xbd/0x150 [dm_mod]
[ÂÂ656.700888][ T3384]ÂÂ? dm_wq_work+0x1b0/0x1b0 [dm_mod]
[ÂÂ656.706073][ T3384]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ656.710821][ T3384]ÂÂgeneric_make_request+0x179/0x4a0
[ÂÂ656.715917][ T3384]ÂÂ? blk_queue_exit+0xc0/0xc0
[ÂÂ656.720489][ T3384]ÂÂ? __unlock_page_memcg+0x4f/0x90
[ÂÂ656.725495][ T3384]ÂÂ? unlock_page_memcg+0x1f/0x30
[ÂÂ656.730329][ T3384]ÂÂsubmit_bio+0xaa/0x270
[ÂÂ656.734466][ T3384]ÂÂ? generic_make_request+0x4a0/0x4a0
[ÂÂ656.739739][ T3384]ÂÂ__swap_writepage+0x8f5/0xba0
[ÂÂ656.744484][ T3384]ÂÂ? __x64_sys_madvise.cold.0+0x22/0x22
[ÂÂ656.749931][ T3384]ÂÂ? generic_swapfile_activate+0x2a0/0x2a0
[ÂÂ656.755638][ T3384]ÂÂ? do_raw_spin_lock+0x118/0x1d0
[ÂÂ656.760559][ T3384]ÂÂ? rwlock_bug.part.0+0x60/0x60
[ÂÂ656.765393][ T3384]ÂÂ? page_swapcount+0x68/0xc0
[ÂÂ656.769967][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.774713][ T3384]ÂÂ? do_raw_spin_unlock+0xa8/0x140
[ÂÂ656.779724][ T3384]ÂÂ? __frontswap_store+0x103/0x2b0
[ÂÂ656.784735][ T3384]ÂÂswap_writepage+0x65/0xb0
[ÂÂ656.789134][ T3384]ÂÂpageout.isra.4+0x3e5/0xa00
[ÂÂ656.793707][ T3384]ÂÂ? shrink_slab+0x440/0x440
[ÂÂ656.798192][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.802939][ T3384]ÂÂshrink_page_list+0x159f/0x2650
[ÂÂ656.807860][ T3384]ÂÂ? page_evictable+0x150/0x150
[ÂÂ656.812606][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.817352][ T3384]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ656.822185][ T3384]ÂÂ? shrink_inactive_list+0x2ea/0x770
[ÂÂ656.827456][ T3384]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ656.832202][ T3384]ÂÂ? do_raw_spin_lock+0x118/0x1d0
[ÂÂ656.837126][ T3384]ÂÂ? rwlock_bug.part.0+0x60/0x60
[ÂÂ656.841959][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.846706][ T3384]ÂÂ? do_raw_spin_unlock+0xa8/0x140
[ÂÂ656.851715][ T3384]ÂÂshrink_inactive_list+0x373/0x770
[ÂÂ656.856812][ T3384]ÂÂ? move_pages_to_lru+0xb60/0xb60
[ÂÂ656.861820][ T3384]ÂÂ? shrink_node_memcg+0xcfa/0x1560
[ÂÂ656.866917][ T3384]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ656.871665][ T3384]ÂÂ? find_next_bit+0x2c/0xa0
[ÂÂ656.876151][ T3384]ÂÂshrink_node_memcg+0x4ff/0x1560
[ÂÂ656.881075][ T3384]ÂÂ? shrink_active_list+0xa10/0xa10
[ÂÂ656.886173][ T3384]ÂÂ? dev_ifsioc+0xb0/0x4d0
[ÂÂ656.890485][ T3384]ÂÂ? mem_cgroup_iter+0x18e/0x840
[ÂÂ656.895319][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.900066][ T3384]ÂÂ? mem_cgroup_protected+0x20f/0x260
[ÂÂ656.905334][ T3384]ÂÂshrink_node+0x1d3/0xa30
[ÂÂ656.909644][ T3384]ÂÂ? shrink_node_memcg+0x1560/0x1560
[ÂÂ656.914828][ T3384]ÂÂ? ktime_get+0x93/0x110
[ÂÂ656.919050][ T3384]ÂÂdo_try_to_free_pages+0x22f/0x820
[ÂÂ656.924146][ T3384]ÂÂ? shrink_node+0xa30/0xa30
[ÂÂ656.928632][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.933379][ T3384]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ656.938212][ T3384]ÂÂtry_to_free_pages+0x242/0x4d0
[ÂÂ656.943046][ T3384]ÂÂ? do_try_to_free_pages+0x820/0x820
[ÂÂ656.948318][ T3384]ÂÂ__alloc_pages_nodemask+0x9ce/0x1bc0
[ÂÂ656.953677][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.958424][ T3384]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ656.963697][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.968443][ T3384]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ656.973277][ T3384]ÂÂ? do_anonymous_page+0x343/0xe30
[ÂÂ656.978288][ T3384]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ656.983035][ T3384]ÂÂ? __count_memcg_events+0x8b/0x1c0
[ÂÂ656.988218][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ656.992966][ T3384]ÂÂ? __lru_cache_add+0x122/0x160
[ÂÂ656.997802][ T3384]ÂÂalloc_pages_vma+0x89/0x2c0
[ÂÂ657.002375][ T3384]ÂÂdo_anonymous_page+0x3e1/0xe30
[ÂÂ657.007211][ T3384]ÂÂ? __update_load_avg_cfs_rq+0x2c/0x490
[ÂÂ657.012743][ T3384]ÂÂ? finish_fault+0x120/0x120
[ÂÂ657.017314][ T3384]ÂÂ? alloc_pages_vma+0x21e/0x2c0
[ÂÂ657.022148][ T3384]ÂÂhandle_pte_fault+0x457/0x12c0
[ÂÂ657.026984][ T3384]ÂÂ__handle_mm_fault+0x79a/0xa50
[ÂÂ657.031819][ T3384]ÂÂ? vmf_insert_mixed_mkwrite+0x20/0x20
[ÂÂ657.037267][ T3384]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ657.042013][ T3384]ÂÂ? __count_memcg_events+0x8b/0x1c0
[ÂÂ657.047199][ T3384]ÂÂhandle_mm_fault+0x17f/0x370
[ÂÂ657.051863][ T3384]ÂÂ__do_page_fault+0x25b/0x5d0
[ÂÂ657.056521][ T3384]ÂÂdo_page_fault+0x4c/0x2cf
[ÂÂ657.060922][ T3384]ÂÂ? page_[ÂÂ659.105948][ T3124] kworker/2:1H: page
allocation failure: order:0, mode:0xa20(GFP_ATOMIC),
nodemask=(null),cpuset=/,mems_allowed=0,4
[ÂÂ659.106045][ T1598] kworker/10:1H: page allocation failure: order:0,
mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0,4
[ÂÂ659.118049][ T3124] CPU: 2 PID: 3124 Comm: kworker/2:1H Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.2.0-next-20190711+ #3
[ÂÂ659.137325][ÂÂT762] ODEBUG: Out of memory. ODEBUG disabled
[ÂÂ659.140015][ T3124] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 06/24/2019
[ÂÂ659.140032][ T3124] Workqueue: kblockd blk_mq_run_work_fn
[ÂÂ659.160266][ T3124] Call Trace:
[ÂÂ659.163442][ T3124]ÂÂdump_stack+0x62/0x9a
[ÂÂ659.167487][ T3124]ÂÂwarn_alloc.cold.45+0x8a/0x12a
[ÂÂ659.172315][ T3124]ÂÂ? zone_watermark_ok_safe+0x1a0/0x1a0
[ÂÂ659.177756][ T3124]ÂÂ? __read_once_size_nocheck.constprop.2+0x10/0x10
[ÂÂ659.184252][ T3124]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.190658][ T3124]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.197060][ T3124]ÂÂ? __isolate_free_page+0x390/0x390
[ÂÂ659.202239][ T3124]ÂÂ__alloc_pages_nodemask+0x1aab/0x1bc0
[ÂÂ659.207680][ T3124]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ659.212949][ T3124]ÂÂ? stack_trace_save+0x87/0xb0
[ÂÂ659.217689][ T3124]ÂÂ? freezing_slow_path.cold.1+0x35/0x35
[ÂÂ659.223219][ T3124]ÂÂ? __kasan_kmalloc.part.0+0x81/0xc0
[ÂÂ659.228485][ T3124]ÂÂ? __kasan_kmalloc.part.0+0x44/0xc0
[ÂÂ659.233750][ T3124]ÂÂ? __kasan_kmalloc.constprop.1+0xac/0xc0
[ÂÂ659.239451][ T3124]ÂÂ? kasan_slab_alloc+0x11/0x20
[ÂÂ659.244196][ T3124]ÂÂ? kmem_cache_alloc+0x17a/0x400
[ÂÂ659.249113][ T3124]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ659.253418][ T3124]ÂÂ? alloc_iova_fast+0x47/0xba
[ÂÂ659.258073][ T3124]ÂÂ? dma_ops_alloc_iova.isra.5+0x86/0xa0
[ÂÂ659.263603][ T3124]ÂÂ? map_sg+0x99/0x2f0
[ÂÂ659.267558][ T3124]ÂÂ? scsi_dma_map+0xc6/0x160
[ÂÂ659.272042][ T3124]ÂÂ? pqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ659.280020][ T3124]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.286421][ T3124]ÂÂ? scsi_queue_rq+0x7c6/0x1280
[ÂÂ659.291163][ T3124]ÂÂ? ftrace_graph_ret_addr+0x2a/0xb0
[ÂÂ659.296340][ T3124]ÂÂ? stack_trace_save+0x87/0xb0
[ÂÂ659.301081][ T3124]ÂÂalloc_pages_current+0x9c/0x110
[ÂÂ659.305998][ T3124]ÂÂallocate_slab+0x351/0x11f0
[ÂÂ659.310564][ T3124]ÂÂnew_slab+0x46/0x70
[ÂÂ659.314433][ T3124]ÂÂ___slab_alloc+0x5d4/0x9c0
[ÂÂ659.318913][ T3124]ÂÂ? should_fail+0x107/0x3bc
[ÂÂ659.323393][ T3124]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ659.327700][ T3124]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ659.332441][ T3124]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ659.337183][ T3124]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ659.341487][ T3124]ÂÂ__slab_alloc+0x12/0x20
[ÂÂ659.345704][ T3124]ÂÂ? __slab_alloc+0x12/0x20
[ÂÂ659.350096][ T3124]ÂÂkmem_cache_alloc+0x32a/0x400
[ÂÂ659.354838][ T3124]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ659.359580][ T3124]ÂÂ? do_raw_spin_unlock+0xa8/0x140
[ÂÂ659.364585][ T3124]ÂÂalloc_iova+0x33/0x210
[ÂÂ659.368714][ T3124]ÂÂ? iova_rcache_get+0x1a1/0x300
[ÂÂ659.373545][ T3124]ÂÂalloc_iova_fast+0x47/0xba
[ÂÂ659.378026][ T3124]ÂÂdma_ops_alloc_iova.isra.5+0x86/0xa0
[ÂÂ659.383381][ T3124]ÂÂmap_sg+0x99/0x2f0
[ÂÂ659.387161][ T3124]ÂÂscsi_dma_map+0xc6/0x160
[ÂÂ659.391470][ T3124]ÂÂpqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ659.399274][ T3124]ÂÂ? pqi_alloc_io_request+0x11e/0x140 [smartpqi]
[ÂÂ659.405507][ T3124]ÂÂpqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.411733][ T3124]ÂÂ? scsi_init_io+0x102/0x150
[ÂÂ659.416306][ T3124]ÂÂ? sd_setup_read_write_cmnd+0x6e9/0xa90 [sd_mod]
[ÂÂ659.422713][ T3124]ÂÂ? pqi_event_worker+0xdf0/0xdf0 [smartpqi]
[ÂÂ659.428593][ T3124]ÂÂ? sd_init_command+0x88b/0x930 [sd_mod]
[ÂÂ659.434211][ T3124]ÂÂ? blk_add_timer+0xd7/0x110
[ÂÂ659.438780][ T3124]ÂÂscsi_queue_rq+0x7c6/0x1280
[ÂÂ659.443350][ T3124]ÂÂblk_mq_dispatch_rq_list+0x9d3/0xba0
[ÂÂ659.448702][ T3124]ÂÂ? blk_mq_flush_busy_ctxs+0x1c5/0x450
[ÂÂ659.454145][ T3124]ÂÂ? blk_mq_get_driver_tag+0x290/0x290
[ÂÂ659.459498][ T3124]ÂÂ? __lock_acquire.isra.13+0x430/0x830
[ÂÂ659.464938][ T3124]ÂÂblk_mq_sched_dispatch_requests+0x2f4/0x300
[ÂÂ659.470903][ T3124]ÂÂ? blk_mq_sched_restart+0x60/0x60
[ÂÂ659.475993][ T3124]ÂÂ__blk_mq_run_hw_queue+0x156/0x230
[ÂÂ659.481172][ T3124]ÂÂ? hctx_lock+0xc0/0xc0
[ÂÂ659.485301][ T3124]ÂÂ? process_one_work+0x426/0xa70
[ÂÂ659.490217][ T3124]ÂÂblk_mq_run_work_fn+0x3b/0x40
[ÂÂ659.494959][ T3124]ÂÂprocess_one_work+0x53b/0xa70
[ÂÂ659.499703][ T3124]ÂÂ? pwq_dec_nr_in_flight+0x170/0x170
[ÂÂ659.504967][ T3124]ÂÂworker_thread+0x63/0x5b0
[ÂÂ659.509361][ T3124]ÂÂkthread+0x1df/0x200
[ÂÂ659.513316][ T3124]ÂÂ? process_one_work+0xa70/0xa70
[ÂÂ659.518231][ T3124]ÂÂ? kthread_park+0xc0/0xc0
[ÂÂ659.522625][ T3124]ÂÂret_from_fork+0x22/0x40
[ÂÂ659.526937][ T1598] CPU: 10 PID: 1598 Comm: kworker/10:1H Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.2.0-next-20190711+ #3
[ÂÂ659.526991][ T3124] Mem-Info:
[ÂÂ659.536921][ T1598] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 06/24/2019
[ÂÂ659.536934][ T1598] Workqueue: kblockd blk_mq_run_work_fn
[ÂÂ659.540067][ T3124] active_anon:4662210 inactive_anon:359358
isolated_anon:2005
[ÂÂ659.540067][ T3124]ÂÂactive_file:10032 inactive_file:12947 isolated_file:0
[ÂÂ659.540067][ T3124]ÂÂunevictable:0 dirty:12 writeback:0 unstable:0
[ÂÂ659.540067][ T3124]ÂÂslab_reclaimable:71207 slab_unreclaimable:1252996
[ÂÂ659.540067][ T3124]ÂÂmapped:17530 shmem:1850 pagetables:11491 bounce:0
[ÂÂ659.540067][ T3124]ÂÂfree:54096 free_pcp:5994 free_cma:84
[ÂÂ659.549192][ T1598] Call Trace:
[ÂÂ659.549203][ T1598]ÂÂdump_stack+0x62/0x9a
[ÂÂ659.554639][ T3124] Node 0 active_anon:2246440kB inactive_anon:572540kB
active_file:19500kB inactive_file:19016kB unevictable:0kB isolated(anon):7708kB
isolated(file):0kB mapped:24840kB dirty:8kB writeback:0kB shmem:1372kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1689600kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
[ÂÂ659.593619][ T1598]ÂÂwarn_alloc.cold.45+0x8a/0x12a
[ÂÂ659.596785][ T3124] Node 1 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ659.600821][ T1598]ÂÂ? zone_watermark_ok_safe+0x1a0/0x1a0
[ÂÂ659.630195][ T3124] Node 2 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ659.635021][ T1598]ÂÂ? __read_once_size_nocheck.constprop.2+0x10/0x10
[ÂÂ659.661328][ T3124] Node 3 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ659.661337][ T3124] Node 4 active_anon:16402112kB inactive_anon:865180kB
active_file:20600kB inactive_file:32712kB unevictable:0kB isolated(anon):304kB
isolated(file):0kB mapped:45216kB dirty:40kB writeback:12kB shmem:6028kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 15167488kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
[ÂÂ659.666778][ T1598]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.693086][ T3124] Node 5 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ659.693096][ T3124] Node 6 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ659.699583][ T1598]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.725894][ T3124] Node 7 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ659.755524][ T1598]ÂÂ? __isolate_free_page+0x390/0x390
[ÂÂ659.761953][ T3124] Node 0 DMA free:15908kB min:24kB low:36kB high:48kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB
kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB
[ÂÂ659.788234][ T1598]ÂÂ__alloc_pages_nodemask+0x1aab/0x1bc0
[ÂÂ659.814544][ T3124] lowmem_reserve[]: 0 1532 19982 19982 19982
[ÂÂ659.820945][ T1598]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ659.847287][ T3124] Node 0 DMA32 free:73504kB min:2676kB low:4244kB
high:5812kB active_anon:1190128kB inactive_anon:362496kB active_file:0kB
inactive_file:0kB unevictable:0kB writepending:0kB present:1923080kB
managed:1634348kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
free_pcp:1432kB local_pcp:0kB free_cma:0kB
[ÂÂ659.852428][ T1598]ÂÂ? stack_trace_save+0x87/0xb0
[ÂÂ659.852435][ T1598]ÂÂ? freezing_slow_path.cold.1+0x35/0x35
[ÂÂ659.879003][ T3124] lowmem_reserve[]: 0 0 18450 18450 18450
[ÂÂ659.884446][ T1598]ÂÂ? __kasan_kmalloc.part.0+0x81/0xc0
[ÂÂ659.890346][ T3124] Node 0 Normal free:47760kB min:137264kB low:156156kB
high:175048kB active_anon:1056208kB inactive_anon:209672kB active_file:19456kB
inactive_file:18996kB unevictable:0kB writepending:0kB present:27262976kB
managed:18893712kB mlocked:0kB kernel_stack:22240kB pagetables:10064kB
bounce:0kB free_pcp:9340kB local_pcp:164kB free_cma:0kB
[ÂÂ659.895574][ T1598]ÂÂ? __kasan_kmalloc.part.0+0x44/0xc0
[ÂÂ659.895581][ T1598]ÂÂ? __kasan_kmalloc.constprop.1+0xac/0xc0
[ÂÂ659.924420][ T3124] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ659.929163][ T1598]ÂÂ? kasan_slab_alloc+0x11/0x20
[ÂÂ659.929170][ T1598]ÂÂ? kmem_cache_alloc+0x17a/0x400
[ÂÂ659.934724][ T3124] Node 4 Normal free:72728kB min:234904kB low:267232kB
high:299560kB active_anon:16401776kB inactive_anon:865580kB active_file:20596kB
inactive_file:32692kB unevictable:0kB writepending:40kB present:33538048kB
managed:32332156kB mlocked:0kB kernel_stack:23040kB pagetables:35900kB
bounce:0kB free_pcp:12956kB local_pcp:24kB free_cma:336kB
[ÂÂ659.940301][ T1598]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ659.940307][ T1598]ÂÂ? alloc_iova_fast+0x47/0xba
[ÂÂ659.945563][ T3124] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ659.976773][ T1598]ÂÂ? dma_ops_alloc_iova.isra.5+0x86/0xa0
[ÂÂ659.976780][ T1598]ÂÂ? map_sg+0x99/0x2f0
[ÂÂ659.982039][ T3124] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U)
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ÂÂ659.987736][ T1598]ÂÂ? scsi_dma_map+0xc6/0x160
[ÂÂ659.987747][ T1598]ÂÂ? pqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ659.992300][ T3124] Node 0 DMA32: 0*4kB 0*8kB 2*16kB (M) 5*32kB (UM) 17*64kB
(UM) 8*128kB (UM) 12*256kB (UM) 11*512kB (UM) 10*1024kB (UM) 2*2048kB (UM)
12*4096kB (M) = 74496kB
[ÂÂ659.997045][ T1598]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ659.997051][ T1598]ÂÂ? scsi_queue_rq+0x7c6/0x1280
[ÂÂ660.001958][ T3124] Node 0 Normal: 0*4kB 0*8kB 198*16kB (MEH) 356*32kB (ME)
83*64kB (UME) 15*128kB (UME) 101*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB =
47648kB
[ÂÂ660.033521][ T1598]ÂÂ? ftrace_graph_ret_addr+0x2a/0xb0
[ÂÂ660.033528][ T1598]ÂÂ? stack_trace_save+0x87/0xb0
[ÂÂ660.037828][ T3124] Node 4 Normal: 0*4kB 0*8kB 211*16kB (UME) 441*32kB (UME)
449*64kB (UME) 71*128kB (ME) 62*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB =
71184kB
[ÂÂ660.042481][ T1598]ÂÂalloc_pages_current+0x9c/0x110
[ÂÂ660.047042][ T3124] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[ÂÂ660.052569][ T1598]ÂÂallocate_slab+0x351/0x11f0
[ÂÂ660.056516][ T3124] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ÂÂ660.056521][ T3124] Node 4 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[ÂÂ660.070694][ T1598]ÂÂnew_slab+0x46/0x70
[ÂÂ660.075169][ T3124] Node 4 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ÂÂ660.083141][ T1598]ÂÂ___slab_alloc+0x5d4/0x9c0
[ÂÂ660.098879][ T3124] 26058 total pagecache pages
[ÂÂ660.098894][ T3124] 1298 pages in swap cache
[ÂÂ660.105279][ T1598]ÂÂ? should_fail+0x107/0x3bc
[ÂÂ660.105285][ T1598]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ660.110020][ T3124] Swap cache stats: add 2607, delete 1311, find 0/1
[ÂÂ660.110024][ T3124] Free swapÂÂ= 32919548kB
[ÂÂ660.124719][ T1598]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ660.124725][ T1598]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ660.129894][ T3124] Total swap = 32952316kB
[ÂÂ660.129899][ T3124] 15685025 pages RAM
[ÂÂ660.134637][ T1598]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ660.149328][ T3124] 0 pages HighMem/MovableOnly
[ÂÂ660.149332][ T3124] 2465994 pages reserved
[ÂÂ660.154245][ T1598]ÂÂ__slab_alloc+0x12/0x20
[ÂÂ660.154252][ T1598]ÂÂ? __slab_alloc+0x12/0x20
[ÂÂ660.163701][ T3124] 16384 pages cma reserved
[ÂÂ660.163763][ T3124] SLUB: Unable to allocate memory on node -1,
gfp=0xa20(GFP_ATOMIC)
[ÂÂ660.168269][ T1598]ÂÂkmem_cache_alloc+0x32a/0x400
[ÂÂ660.168276][ T1598]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ660.177465][ T3124]ÂÂÂcache: iommu_iova, object size: 40, buffer size: 448,
default order: 0, min order: 0
[ÂÂ660.177470][ T3124]ÂÂÂnode 0: slabs: 10580, objs: 95220, free: 0
[ÂÂ660.186924][ T1598]ÂÂ? do_raw_spin_unlock+0xa8/0x140
[ÂÂ660.186930][ T1598]ÂÂalloc_iova+0x33/0x210
[ÂÂ660.190792][ T3124]ÂÂÂnode 4: slabs: 2292, objs: 20628, free: 25
[ÂÂ660.199982][ T1598]ÂÂ? iova_rcache_get+0x1a1/0x300
[ÂÂ660.199989][ T1598]ÂÂalloc_iova_fast+0x47/0xba
[ÂÂ660.204513][ T3124] kworker/2:1H: page allocation failure: order:0,
mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0,4
[ÂÂ660.209026][ T1598]ÂÂdma_ops_alloc_iova.isra.5+0x86/0xa0
[ÂÂ660.351109][ T1598]ÂÂmap_sg+0x99/0x2f0
[ÂÂ660.354891][ T1598]ÂÂ? __debug_object_init+0x412/0x7a0
[ÂÂ660.360070][ T1598]ÂÂscsi_dma_map+0xc6/0x160
[ÂÂ660.364381][ T1598]ÂÂpqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ660.372184][ T1598]ÂÂ? pqi_alloc_io_request+0x11e/0x140 [smartpqi]
[ÂÂ660.378415][ T1598]ÂÂpqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ660.384644][ T1598]ÂÂ? scsi_init_io+0x102/0x150
[ÂÂ660.389217][ T1598]ÂÂ? sd_setup_read_write_cmnd+0x6e9/0xa90 [sd_mod]
[ÂÂ660.395622][ T1598]ÂÂ? pqi_event_worker+0xdf0/0xdf0 [smartpqi]
[ÂÂ660.401503][ T1598]ÂÂ? sd_init_command+0x88b/0x930 [sd_mod]
[ÂÂ660.407119][ T1598]ÂÂ? blk_add_timer+0xd7/0x110
[ÂÂ660.411686][ T1598]ÂÂscsi_queue_rq+0x7c6/0x1280
[ÂÂ660.416252][ T1598]ÂÂblk_mq_dispatch_rq_list+0x9d3/0xba0
[ÂÂ660.421604][ T1598]ÂÂ? blk_mq_flush_busy_ctxs+0x1c5/0x450
[ÂÂ660.427045][ T1598]ÂÂ? blk_mq_get_driver_tag+0x290/0x290
[ÂÂ660.432396][ T1598]ÂÂ?
__lock_acquire.isra.13+0xT3124]ÂÂ__blk_mq_run_hw_queue+0x156/0x230
[ÂÂ660.822569][ T3124]ÂÂ? hctx_lock+0xc0/0xc0
[ÂÂ660.826700][ T3124]ÂÂ? process_one_work+0x426/0xa70
[ÂÂ660.831617][ T3124]ÂÂblk_mq_run_work_fn+0x3b/0x40
[ÂÂ660.836358][ T3124]ÂÂprocess_one_work+0x53b/0xa70
[ÂÂ660.841100][ T3124]ÂÂ? pwq_dec_nr_in_flight+0x170/0x170
[ÂÂ660.846365][ T3124]ÂÂworker_thread+0x63/0x5b0
[ÂÂ660.850756][ T3124]ÂÂkthread+0x1df/0x200
[ÂÂ660.854712][ T3124]ÂÂ? process_one_work+0xa70/0xa70
[ÂÂ660.859626][ T3124]ÂÂ? kthread_park+0xc0/0xc0
[ÂÂ660.864021][ T3124]ÂÂret_from_fork+0x22/0x40
[ÂÂ660.868328][ T3124] warn_alloc_show_mem: 1 callbacks suppressed
[ÂÂ660.868332][ T1598] CPU: 10 PID: 1598 Comm: kworker/10:1H Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.2.0-next-20190711+ #3
[ÂÂ660.868335][ T3124] Mem-Info:
[ÂÂ660.868485][ T3124] active_anon:4662011 inactive_anon:359383
isolated_anon:2155
[ÂÂ660.868485][ T3124]ÂÂactive_file:10012 inactive_file:12922 isolated_file:0
[ÂÂ660.868485][ T3124]ÂÂunevictable:0 dirty:12 writeback:0 unstable:0
[ÂÂ660.868485][ T3h:175048kB active_anon:1056208kB inactive_anon:209448kB
active_file:19452kB inactive_file:18996kB unevictable:0kB writepending:0kB
present:27262976kB managed:18893712kB mlocked:0kB kernel_stack:22240kB
pagetables:10064kB bounce:0kB free_pcp:8784kB local_pcp:164kB free_cma:0kB
[ÂÂ661.222532][ T1598]ÂÂ? kernel_poison_pages.cold.2+0x8c/0x8c
[ÂÂ661.228397][ T3124] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ661.233138][ T1598]ÂÂ? vprintk_default+0x1f/0x30
[ÂÂ661.233146][ T1598]ÂÂalloc_pages_current+0x9c/0x110
[ÂÂ661.238174][ T3124] Node 4 Normal free:71384kB min:234904kB low:267232kB
high:299560kB active_anon:16401776kB inactive_anon:865588kB active_file:20596kB
inactive_file:32692kB unevictable:0kB writepending:40kB present:33538048kB
managed:32332156kB mlocked:0kB kernel_stack:23040kB pagetables:35900kB
bounce:0kB free_pcp:12872kB local_pcp:24kB free_cma:336kB
[ÂÂ661.266900][ T1598]ÂÂallocate_slab+0x351/0x11f0
[ÂÂ661.266905][ T1598]ÂÂnew_slab+0x46/0x70
[ÂÂ661.271461][ T3124] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ661.275941][ T1598]ÂÂ___slab_alloc+0x5d4/0x9c0
[ÂÂ661.275948][ T1598]ÂÂ? should0
[ÂÂ661.543007][ T3132]ÂÂÂcache: iommu_iova, object size: 40, buffer size: 448,
default order: 0, min order: 0
[ÂÂ661.543011][ T3203]ÂÂÂnode 0: slabs: 10582, objs: 95238, free: 7
[ÂÂ661.543016][ T3132]ÂÂÂnode 0: slabs: 10582, objs: 95238, free: 7
[ÂÂ661.543020][ T3203]ÂÂÂnode 4: slabs: 2293, objs: 20637, free: 30
[ÂÂ661.543026][ T3132]ÂÂÂnode 4: slabs: 2293, objs: 20637, free: 30
[ÂÂ661.543040][ T3203] SLUB: Unable to allocate memory on node -1,
gfp=0xa20(GFP_ATOMIC)
[ÂÂ661.543046][ T3203]ÂÂÂcache: iommu_iova, object size: 40, buffer size: 448,
default order: 0, min order: 0
[ÂÂ661.543052][ T3203]ÂÂÂnode 0: slabs: 10582, objs: 95238, free: 7
[ÂÂ661.543057][ T3132] SLUB: Unable to allocate memory on node -1,
gfp=0xa20(GFP_ATOMIC)
[ÂÂ661.543061][ T3203]ÂÂÂnode 4: slabs: 2293, objs: 20637, free: 30
[ÂÂ661.543066][ T3132]ÂÂÂcache: iommu_iova, object size: 40, buffer size: 448,
default order: 0, min order: 0
[ÂÂ661.543072][ T3132]ÂÂÂnode 0: slabs: 10582, objs: 95238, free: 7
[ÂÂ661.543078][ T3132]ÂÂÂnode 4: slabs: 2293, objs: 20637, free: 30
[ÂÂ661.543544][ T3205] SLUB: Unable to allocnevictable:0kB isolated(anon):352kB
isolated(file):0kB mapped:45056kB dirty:40kB writeback:52kB shmem:6028kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 15167488kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
[ÂÂ662.181289][ T1598] Node 5 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ662.207607][ T3209]ÂÂ? __read_once_size_nocheck.constprop.2+0x10/0x10
[ÂÂ662.212434][ T1598] Node 6 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ662.238751][ T3209]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ662.244187][ T1598] Node 7 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anoÂÂalloc_iova_fast+0x47/0xba
[ÂÂ662.835750][ T3209]ÂÂdma_ops_alloc_iova.isra.5+0x86/0xa0
[ÂÂ662.841103][ T3209]ÂÂmap_sg+0x99/0x2f0
[ÂÂ662.844886][ T3209]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ662.849627][ T3209]ÂÂscsi_dma_map+0xc6/0x160
[ÂÂ662.853938][ T3209]ÂÂpqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ662.861740][ T3209]ÂÂ? pqi_alloc_io_request+0x11e/0x140 [smartpqi]
[ÂÂ662.867971][ T3209]ÂÂpqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ662.874198][ T3209]ÂÂ? scsi_init_io+0x102/0x150
[ÂÂ662.878768][ T3209]ÂÂ? sd_setup_read_write_cmnd+0x6e9/0xa90 [sd_mod]
[ÂÂ662.885176][ T3209]ÂÂ? pqi_event_worker+0xdf0/0xdf0 [smartpqi]
[ÂÂ662.891055][ T3209]ÂÂ? sd_init_command+0x88b/0x930 [sd_mod]
[ÂÂ662.896672][ T3209]ÂÂ? blk_add_timer+0xd7/0x110
[ÂÂ662.901240][ T3209]ÂÂscsi_queue_rq+0x7c6/0x1280
[ÂÂ662.905807][ T3209]ÂÂblk_mq_dispatch_rq_list+0x9d3/0xba0
[ÂÂ662.911159][ T3209]ÂÂ? blk_mq_flush_busy_ctxs+0x1c5/0x450
[ÂÂ662.916601][ T3209]ÂÂ? blk_mq_get_driver_tag+0x290/0x290
[ÂÂ662.921953][ T3209]ÂÂ? __lock_acquire.isra.13+0x430/0x830
[ÂÂ662.927394][ T3209]ÂÂblk_mq_sched_diag+0x290/0x290
[ÂÂ663.313403][ T3146]ÂÂ? __lock_acquire.isra.13+0x430/0x830
[ÂÂ663.318844][ T3146]ÂÂblk_mq_sched_dispatch_requests+0x2f4/0x300
[ÂÂ663.324807][ T3146]ÂÂ? blk_mq_sched_restart+0x60/0x60
[ÂÂ663.329898][ T3146]ÂÂ__blk_mq_run_hw_queue+0x156/0x230
[ÂÂ663.335076][ T3146]ÂÂ? hctx_lock+0xc0/0xc0
[ÂÂ663.339211][ T3146]ÂÂ? process_one_work+0x426/0xa70
[ÂÂ663.344128][ T3146]ÂÂblk_mq_run_work_fn+0x3b/0x40
[ÂÂ663.348870][ T3146]ÂÂprocess_one_work+0x53b/0xa70
[ÂÂ663.353613][ T3146]ÂÂ? pwq_dec_nr_in_flight+0x170/0x170
[ÂÂ663.358880][ T3146]ÂÂworker_thread+0x63/0x5b0
[ÂÂ663.363277][ T3146]ÂÂkthread+0x1df/0x200
[ÂÂ663.367233][ T3146]ÂÂ? process_one_work+0xa70/0xa70
[ÂÂ663.372148][ T3146]ÂÂ? kthread_park+0xc0/0xc0
[ÂÂ663.376543][ T3146]ÂÂret_from_fork+0x22/0x40
[ÂÂ663.380848][ T3146] warn_alloc_show_mem: 1 callbacks suppressed
[ÂÂ663.380855][ T3123] CPU: 1 PID: 3123 Comm: kworker/1:1H Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.2.0-next-20190711+ #3
[ÂÂ663.380857][ T3146] Mem-Info:
[ÂÂ663.381000][ T3146] active_anon:4654271 inactive_anon:367023
isolated_anon:2263
[ÂÂ663.381000T3123]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ663.744691][ T3146] Node 0 Normal free:74264kB min:137264kB low:156156kB
high:175048kB active_anon:1055816kB inactive_anon:209292kB active_file:19416kB
inactive_file:18964kB unevictable:0kB writepending:248kB present:27262976kB
managed:18893712kB mlocked:0kB kernel_stack:22240kB pagetables:10064kB
bounce:0kB free_pcp:9356kB local_pcp:124kB free_cma:0kB
[ÂÂ663.750101][ T3123]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ663.778942][ T3146] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ663.783688][ T3123]ÂÂ? do_raw_spin_lock+0x118/0x1d0
[ÂÂ663.789326][ T3146] Node 4 Normal free:81632kB min:234904kB low:267232kB
high:299560kB active_anon:16368972kB inactive_anon:898504kB active_file:20548kB
inactive_file:32468kB unevictable:0kB writepending:104kB present:33538048kB
managed:32332156kB mlocked:0kB kernel_stack:23040kB pagetables:35900kB
bounce:0kB free_pcp:11372kB local_pcp:160kB free_cma:0kB
[ÂÂ663.794556][ T3123]ÂÂ? rwlock_bug.part.0+0x60/0x60
[ÂÂ663.794563][ T3123]ÂÂ? get_partial_node+0x48/0x540
[ÂÂ663.825936][ T3146] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ663.830678][ T3123]ÂÂÂ#3
[ÂÂ664.269661][ T3202] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 06/24/2019
[ÂÂ664.278993][ T3202] Workqueue: kblockd blk_mq_run_work_fn
[ÂÂ664.284453][ T3202] Call Trace:
[ÂÂ664.287655][ T3202]ÂÂdump_stack+0x62/0x9a
[ÂÂ664.291721][ T3202]ÂÂwarn_alloc.cold.45+0x8a/0x12a
[ÂÂ664.296577][ T3202]ÂÂ? zone_watermark_ok_safe+0x1a0/0x1a0
[ÂÂ664.302044][ T3202]ÂÂ? __read_once_size_nocheck.constprop.2+0x10/0x10
[ÂÂ664.308564][ T3202]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ664.314996][ T3202]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ664.321420][ T3202]ÂÂ? __isolate_free_page+0x390/0x390
[ÂÂ664.326613][ T3202]ÂÂ__alloc_pages_nodemask+0x1aab/0x1bc0
[ÂÂ664.332062][ T3202]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ664.337345][ T3202]ÂÂ? stack_trace_save+0x87/0xb0
[ÂÂ664.342103][ T3202]ÂÂ? freezing_slow_path.cold.1+0x35/0x35
[ÂÂ664.347647][ T3202]ÂÂ? __kasan_kmalloc.part.0+0x81/0xc0
[ÂÂ664.352925][ T3202]ÂÂ? __kasan_kmalloc.part.0+0x44/0xc0
[ÂÂ664.358204][ T3202]ÂÂ? __kasan_kmalloc.constprop.1+0xac/0xc0
[ÂÂ664.363922][ hmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB
all_unreclaimable? no
[ÂÂ664.759472][ T3127]ÂÂ? __read_once_size_nocheck.constprop.2+0x10/0x10
[ÂÂ664.759508][ T3127]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ664.785836][ T3202] Node 4 active_anon:15362196kB inactive_anon:1296156kB
active_file:15052kB inactive_file:17752kB unevictable:0kB isolated(anon):66644kB
isolated(file):112kB mapped:30596kB dirty:0kB writeback:3968kB shmem:1080kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 14735360kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
[ÂÂ664.789031][ T3127]ÂÂ? pqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ664.793056][ T3202] Node 5 active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
mapped:0kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ÂÂ664.819386][ T3127]ÂÂ? __isolate_free_page+0x390/0x390
[ÂÂ664.819401][ T3127]ÂÂ__alloc_pages_nodemask+0x1aab/0x1bc0
[ÂÂ664.824245][ T3202] Node 6 active_anon7]ÂÂmap_sg+0x99/0x2f0
[ÂÂ665.159320][ T3202] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ÂÂ665.191157][ T3127]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ665.191176][ T3127]ÂÂscsi_dma_map+0xc6/0x160
[ÂÂ665.195480][ T3202] Node 4 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[ÂÂ665.195490][ T3202] Node 4 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ÂÂ665.200248][ T3127]ÂÂpqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ665.204805][ T3202] 69668 total pagecache pages
[ÂÂ665.209566][ T3127]ÂÂ? pqi_alloc_io_request+0x11e/0x140 [smartpqi]
[ÂÂ665.213886][ T3202] 65404 pages in swap cache
[ÂÂ665.228054][ T3127]ÂÂpqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ665.228074][ T3127]ÂÂ? scsi_init_io+0x102/0x150
[ÂÂ665.232285][ T3202] Swap cache stats: add 486050, delete 428240, find 59/149
[ÂÂ665.232294][ T3202] Free swapÂÂ= 30975484kB
[ÂÂ665.236832][ T3127]ÂÂ? sd_setup_read_write_cmnd+0x6e9/0xa90 [sd_mod]
[ÂÂ665.236858][ T3127]ÂÂ? pqi_event_worker+0xdf0/0xdf0 [smar390
[ÂÂ665.806891][ T3141]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ665.811664][ T3141]ÂÂ? alloc_iova+0x33/0x210
[ÂÂ665.815987][ T3141]ÂÂ__slab_alloc+0x12/0x20
[ÂÂ665.820232][ T3141]ÂÂ? __slab_alloc+0x12/0x20
[ÂÂ665.824654][ T3141]ÂÂkmem_cache_alloc+0x32a/0x400
[ÂÂ665.829413][ T3141]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ665.834179][ T3141]ÂÂ? do_raw_spin_unlock+0xa8/0x140
[ÂÂ665.839221][ T3141]ÂÂalloc_iova+0x33/0x210
[ÂÂ665.843369][ T3141]ÂÂ? iova_rcache_get+0x1a1/0x300
[ÂÂ665.848225][ T3141]ÂÂalloc_iova_fast+0x47/0xba
[ÂÂ665.852736][ T3141]ÂÂdma_ops_alloc_iova.isra.5+0x86/0xa0
[ÂÂ665.858122][ T3141]ÂÂmap_sg+0x99/0x2f0
[ÂÂ665.861957][ T3141]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ665.866759][ T3141]ÂÂscsi_dma_map+0xc6/0x160
[ÂÂ665.871098][ T3141]ÂÂpqi_raid_submit_scsi_cmd_with_io_request+0x1c3/0x470
[smartpqi]
[ÂÂ665.878918][ T3141]ÂÂ? pqi_alloc_io_request+0x11e/0x140 [smartpqi]
[ÂÂ665.885172][ T3141]ÂÂpqi_scsi_queue_command+0x791/0xdd0 [smartpqi]
[ÂÂ665.891435][ T3141]ÂÂ? scsi_init_io+0x102/0x150
[ÂÂ665.896103][ T3141]ÂÂ? sd_setup_read_write_cmnd+0x6e9/0xa90 [sd_mod]
[ÂÂ665.902619][ T3141]ÂÂ? pqie:0kB unevictable:0kB writepending:0kB
present:15996kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ÂÂ666.300385][ T3141] lowmem_reserve[]: 0 1532 19982 19982 19982
[ÂÂ666.306395][ T3141] Node 0 DMA32 free:75568kB min:2676kB low:4244kB
high:5812kB active_anon:749752kB inactive_anon:395332kB active_file:128kB
inactive_file:168kB unevictable:0kB writepending:0kB present:1923080kB
managed:1634348kB mlocked:0kB kernel_stack:0kB pagetables:28kB bounce:0kB
free_pcp:55484kB local_pcp:248kB free_cma:0kB
[ÂÂ666.335894][ T3141] lowmem_reserve[]: 0 0 18450 18450 18450
[ÂÂ666.341762][ T3141] Node 0 Normal free:52856kB min:52716kB low:71608kB
high:90500kB active_anon:1127696kB inactive_anon:80184kB active_file:492kB
inactive_file:656kB unevictable:0kB writepending:2208kB present:27262976kB
managed:18893712kB mlocked:0kB kernel_stack:22240kB pagetables:10372kB
bounce:0kB free_pcp:12848kB local_pcp:36kB free_cma:0kB
[ÂÂ666.372602][ T3141] lowmem_reserve[]: 0 0 0 0 0
[ÂÂ666.377419][ T3141] Node 4 Normal free:234488kB m[ÂÂ685.274656][ T3456]
list_del corruption. prev->next should be ffffea0022b10098, but was
0000000000000000
[ÂÂ685.284254][ T3456] ------------[ cut here ]------------
[ÂÂ685.289616][ T3456] kernel BUG at lib/list_debug.c:53!
[ÂÂ685.294808][ T3456] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN NOPTI
[ÂÂ685.301998][ T3456] CPU: 5 PID: 3456 Comm: oom01 Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.2.0-next-20190711+ #3
[ÂÂ685.311193][ T3456] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 06/24/2019
[ÂÂ685.320485][ T3456] RIP: 0010:__list_del_entry_valid+0x8b/0xb6
[ÂÂ685.326364][ T3456] Code: f1 e0 ff 49 8b 55 08 4c 39 e2 75 2c 5b b8 01 00 00
00 41 5c 41 5d 5d c3 4c 89 e2 48 89 de 48 c7 c7 c0 5a 73 a3 e8 d9 fa bc ff <0f>
0b 48 c7 c7 60 a0 e1 a3 e8 13 52 01 00 4c 89 e6 48 c7 c7 20 5b
[ÂÂ685.345956][ T3456] RSP: 0018:ffff888e0c8a73c0 EFLAGS: 00010082
[ÂÂ685.351920][ T3456] RAX: 0000000000000054 RBX: ffffea0022b10098 RCX:
ffffffffa2d5d708
[ÂÂ685.359807][ T3456] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
ffff8888442bd380
[ÂÂ685.367693][ T3456] RBP: ffff888e0c8a73d8 R08: ffffed1108857a71 R09:
ffffed1108857a70
[ÂÂ685.375577][ T3456] R10: ffffed1108857a70 R11: ffff8888442bd387 R12:
0000000000000000
[ÂÂ685.383462][ T3456] R13: 0000000000000000 R14: ffffea0022b10034 R15:
ffffea0022b10098
[ÂÂ685.391348][ T3456] FS:ÂÂ00007fbe26db4700(0000) GS:ffff888844280000(0000)
knlGS:0000000000000000
[ÂÂ685.400194][ T3456] CS:ÂÂ0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ÂÂ685.406681][ T3456] CR2: 00007fbcabb3f000 CR3: 0000001012e44000 CR4:
00000000001406a0
[ÂÂ685.414563][ T3456] Call Trace:
[ÂÂ685.417736][ T3456]ÂÂdeferred_split_scan+0x337/0x740
[ÂÂ685.422741][ T3456]ÂÂ? split_huge_page_to_list+0xe10/0xe10
[ÂÂ685.428272][ T3456]ÂÂ? __radix_tree_lookup+0x12d/0x1e0
[ÂÂ685.433453][ T3456]ÂÂ? node_tag_get.part.0.constprop.6+0x40/0x40
[ÂÂ685.439505][ T3456]ÂÂdo_shrink_slab+0x244/0x5a0
[ÂÂ685.444071][ T3456]ÂÂshrink_slab+0x253/0x440
[ÂÂ685.448375][ T3456]ÂÂ? unregister_shrinker+0x110/0x110
[ÂÂ685.453551][ T3456]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ685.458291][ T3456]ÂÂ? mem_cgroup_protected+0x20f/0x260
[ÂÂ685.463555][ T3456]ÂÂshrink_node+0x31e/0xa30
[ÂÂ685.467858][ T3456]ÂÂ? shrink_node_memcg+0x1560/0x1560
[ÂÂ685.473036][ T3456]ÂÂ? ktime_get+0x93/0x110
[ÂÂ685.477250][ T3456]ÂÂdo_try_to_free_pages+0x22f/0x820
[ÂÂ685.482338][ T3456]ÂÂ? shrink_node+0xa30/0xa30
[ÂÂ685.486815][ T3456]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ685.491556][ T3456]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ685.496383][ T3456]ÂÂtry_to_free_pages+0x242/0x4d0
[ÂÂ685.501209][ T3456]ÂÂ? do_try_to_free_pages+0x820/0x820
[ÂÂ685.506476][ T3456]ÂÂ__alloc_pages_nodemask+0x9ce/0x1bc0
[ÂÂ685.511826][ T3456]ÂÂ? gfp_pfmemalloc_allowed+0xc0/0xc0
[ÂÂ685.517089][ T3456]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ685.521826][ T3456]ÂÂ? check_chain_key+0x1df/0x2e0
[ÂÂ685.526657][ T3456]ÂÂ? do_anonymous_page+0x343/0xe30
[ÂÂ685.531658][ T3456]ÂÂ? lock_downgrade+0x390/0x390
[ÂÂ685.536399][ T3456]ÂÂ? get_kernel_page+0xa0/0xa0
[ÂÂ685.541050][ T3456]ÂÂ? __lru_cache_add+0x108/0x160
[ÂÂ685.545879][ T3456]ÂÂalloc_pages_vma+0x89/0x2c0
[ÂÂ685.550444][ T3456]ÂÂdo_anonymous_page+0x3e1/0xe30
[ÂÂ685.555271][ T3456]ÂÂ? __update_load_avg_cfs_rq+0x2c/0x490
[ÂÂ685.560796][ T3456]ÂÂ? finish_fault+0x120/0x120
[ÂÂ685.565361][ T3456]ÂÂ? alloc_pages_vma+0x21e/0x2c0
[ÂÂ685.570187][ T3456]ÂÂhandle_pte_fault+0x457/0x12c0
[ÂÂ685.575014][ T3456]ÂÂ__handle_mm_fault+0x79a/0xa50
[ÂÂ685.579841][ T3456]ÂÂ? vmf_insert_mixed_mkwrite+0x20/0x20
[ÂÂ685.585280][ T3456]ÂÂ? kasan_check_read+0x11/0x20
[ÂÂ685.590021][ T3456]ÂÂ? __count_memcg_events+0x8b/0x1c0
[ÂÂ685.595196][ T3456]ÂÂhandle_mm_fault+0x17f/0x370
[ÂÂ685.599850][ T3456]ÂÂ__do_page_fault+0x25b/0x5d0
[ÂÂ685.604501][ T3456]ÂÂdo_page_fault+0x4c/0x2cf
[ÂÂ685.608892][ T3456]ÂÂ? page_fault+0x5/0x20
[ÂÂ685.613019][ T3456]ÂÂpage_fault+0x1b/0x20
[ÂÂ685.617058][ T3456] RIP: 0033:0x410be0
[ÂÂ685.620840][ T3456] Code: 89 de e8 e3 23 ff ff 48 83 f8 ff 0f 84 86 00 00 00
48 89 c5 41 83 fc 02 74 28 41 83 fc 03 74 62 e8 95 29 ff ff 31 d2 48 98 90 <c6>
44 15 00 07 48 01 c2 48 39 d3 7f f3 31 c0 5b 5d 41 5c c3 0f 1f
[ÂÂ68[ÂÂ687.120156][ T3456] Shutting down cpus with NMI
[ÂÂ687.124731][ T3456] Kernel Offset: 0x21800000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ÂÂ687.136389][ T3456] ---[ end Kernel panic - not syncing: Fatal exception ]---