Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc)

[David Hildenbrand]

On 01.06.24 08:01, Yu Zhao wrote:
> On Wed, May 15, 2024 at 4:06 PM Yu Zhao <yuzhao@xxxxxxxxxx> wrote:
> > On Wed, May 15, 2024 at 2:45 PM Erhard Furtner <erhard_f@xxxxxxxxxxx> wrote:
> > > On Wed, 8 May 2024 20:21:11 +0200
> > > Erhard Furtner <erhard_f@xxxxxxxxxxx> wrote:
> > >
> > > > Greetings!
> > > >
> > > > Got that on my dual CPU PowerMac G4 DP shortly after boot. This does not happen every time at bootup though:
> > > >
> > > > [...]
> > > > kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
> > > > CPU: 1 PID: 40 Comm: kswapd0 Not tainted 6.8.9-gentoo-PMacG4 #1
> > > > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > >
> > > Very similar page allocation failure on the same machine on kernel 6.9.0 too. Seems it can easily be provoked by running a memory stressor, e.g. "stress-ng --vm 2 --vm-bytes 1930M --verify -v":
> > >
> > > [...]
> > > kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
> > > CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
> > > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > > Call Trace:
> > > [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
> > > [c1c65960] [c01b6234] warn_alloc+0x100/0x178
> > > [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
> > > [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
> > > [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
> > > [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
> > > [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
> > > [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
> > > [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
> > > [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
> > > [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
> > > [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
> > > [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
> > > [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
> > > [c1c65f50] [c017f270] kswapd+0x228/0x25c
> > > [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
> > > [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 15, objs: 225, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > Mem-Info:
> > > active_anon:340071 inactive_anon:139179 isolated_anon:0
> > >   active_file:8297 inactive_file:2506 isolated_file:0
> > >   unevictable:4 dirty:1 writeback:18
> > >   slab_reclaimable:1377 slab_unreclaimable:7426
> > >   mapped:6804 shmem:112 pagetables:946
> > >   sec_pagetables:0 bounce:0
> > >   kernel_misc_reclaimable:0
> > >   free:1141 free_pcp:7 free_cma:0
> > > Node 0 active_anon:1360284kB inactive_anon:556716kB active_file:33188kB inactive_file:10024kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:27216kB dirty:4kB writeback:72kB shmem:448kB writeback_tmp:0kB kernel_stack:1560kB pagetables:3784kB sec_pagetables:0kB all_unreclaimable? no
> > > DMA free:56kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:635128kB inactive_anon:58260kB active_file:268kB inactive_file:3000kB unevictable:0kB writepending:324kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:28kB free_cma:0kB
> > > lowmem_reserve[]: 0 0 1280 1280
> > > DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> > > 63943 total pagecache pages
> > > 53024 pages in swap cache
> > > Free swap  = 8057248kB
> > > Total swap = 8388604kB
> > > 524288 pages RAM
> > > 327680 pages HighMem/MovableOnly
> > > 9947 pages reserved
> > > warn_alloc: 396 callbacks suppressed
> > > kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
> > > CPU: 1 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
> > > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > > Call Trace:
> > > [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
> > > [c1c65960] [c01b6234] warn_alloc+0x100/0x178
> > > [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
> > > [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
> > > [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
> > > [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
> > > [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
> > > [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
> > > [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
> > > [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
> > > [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
> > > [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
> > > [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
> > > [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
> > > [c1c65f50] [c017f270] kswapd+0x228/0x25c
> > > slab_out_of_memory: 53 callbacks suppressed
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
> > > [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
> > > Mem-Info:
> > > active_anon:351976 inactive_anon:123514 isolated_anon:0
> > >   active_file:4648 inactive_file:2081 isolated_file:0
> > >   unevictable:4 dirty:1 writeback:39
> > >   slab_reclaimable:918 slab_unreclaimable:7222
> > >   mapped:5359 shmem:21 pagetables:940
> > >   sec_pagetables:0 bounce:0
> > >   kernel_misc_reclaimable:0
> > >   free:2563 free_pcp:142 free_cma:0
> > > Node 0 active_anon:1407904kB inactive_anon:494056kB active_file:18592kB inactive_file:8324kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:21436kB dirty:4kB writeback:156kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3760kB sec_pagetables:0kB all_unreclaimable? no
> > > DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:199336kB inactive_anon:491432kB active_file:4612kB inactive_file:5980kB unevictable:0kB writepending:660kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:568kB local_pcp:20kB free_cma:0kB
> > > lowmem_reserve[]: 0 0 1280 1280
> > > DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> > > 45961 total pagecache pages
> > > 39207 pages in swap cache
> > > Free swap  = 8093096kB
> > > Total swap = 8388604kB
> > > 524288 pages RAM
> > > 327680 pages HighMem/MovableOnly
> > > 9947 pages reserved
> > > warn_alloc: 343 callbacks suppressed
> > > kswapd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
> > > CPU: 0 PID: 41 Comm: kswapd0 Not tainted 6.9.0-gentoo-PMacG4 #1
> > > Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
> > > Call Trace:
> > > [c1c65940] [c07926d4] dump_stack_lvl+0x80/0xac (unreliable)
> > > [c1c65960] [c01b6234] warn_alloc+0x100/0x178
> > > [c1c659c0] [c01b661c] __alloc_pages+0x370/0x8d0
> > > [c1c65a80] [c01c4854] __read_swap_cache_async+0xc0/0x1cc
> > > [c1c65ad0] [c01cb580] zswap_writeback_entry+0x50/0x154
> > > [c1c65be0] [c01cb6f4] shrink_memcg_cb+0x70/0xec
> > > [c1c65c10] [c019518c] __list_lru_walk_one+0xa0/0x154
> > > slab_out_of_memory: 59 callbacks suppressed
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: skbuff_small_head, object size: 480, buffer size: 544, default order: 1, min order: 0
> > >    node 0: slabs: 18, objs: 270, free: 0
> > > SLUB: Unable to allocate memory on node -1, gfp=0x820(GFP_ATOMIC)
> > >    cache: kmalloc-rnd-15-2k, object size: 2048, buffer size: 6144, default order: 3, min order: 1
> > >    kmalloc-rnd-15-2k debugging increased min order, use slab_debug=O to disable.
> > >    node 0: slabs: 33, objs: 165, free: 0
> > > [c1c65c70] [c01952a4] list_lru_walk_one+0x64/0x7c
> > > [c1c65ca0] [c01cad00] zswap_shrinker_scan+0xac/0xc4
> > > [c1c65cd0] [c018052c] do_shrink_slab+0x18c/0x304
> > > [c1c65d20] [c0180a40] shrink_slab+0x174/0x260
> > > [c1c65da0] [c017cb0c] shrink_one+0xbc/0x134
> > > [c1c65dd0] [c017e3e4] shrink_node+0x238/0x84c
> > > [c1c65e50] [c017ed38] balance_pgdat+0x340/0x650
> > > [c1c65f50] [c017f270] kswapd+0x228/0x25c
> > > [c1c65fc0] [c006bbac] kthread+0xe4/0xe8
> > > [c1c65ff0] [c0015304] start_kernel_thread+0x10/0x14
> > > Mem-Info:
> > > active_anon:235002 inactive_anon:240975 isolated_anon:0
> > >   active_file:4356 inactive_file:2551 isolated_file:0
> > >   unevictable:4 dirty:7 writeback:19
> > >   slab_reclaimable:1008 slab_unreclaimable:7218
> > >   mapped:5601 shmem:21 pagetables:939
> > >   sec_pagetables:0 bounce:0
> > >   kernel_misc_reclaimable:0
> > >   free:1340 free_pcp:23 free_cma:0
> > > Node 0 active_anon:940008kB inactive_anon:963900kB active_file:17424kB inactive_file:10204kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:22404kB dirty:28kB writeback:76kB shmem:84kB writeback_tmp:0kB kernel_stack:1552kB pagetables:3756kB sec_pagetables:0kB all_unreclaimable? no
> > > DMA free:0kB boost:7756kB min:11208kB low:12068kB high:12928kB reserved_highatomic:0KB active_anon:644060kB inactive_anon:36332kB active_file:5276kB inactive_file:5516kB unevictable:0kB writepending:348kB present:786432kB managed:746644kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:92kB free_cma:0kB
> > > lowmem_reserve[]: 0 0 1280 1280
> > > DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
> > > 116345 total pagecache pages
> > > 109413 pages in swap cache
> > > Free swap  = 7819300kB
> > > Total swap = 8388604kB
> > > 524288 pages RAM
> > > 327680 pages HighMem/MovableOnly
> > > 9947 pages reserved
> > >
> > >
> > > I switched from zstd to lzo as zswap default compressor so zstd does not show up on the dmesg. But the rest looks pretty similar.
> > >
> > > Full dmesg and kernel .config attached.
> > >
> > > Regards,
> > > Erhard
> >
> > Hi Erhard,
> >
> > Thanks for the reports. I'll take a look at them and get back to you
> > in a few days.
>
> Hi Erhard,
>
> The OOM kills on both kernel versions seem to be reasonable to me.
>
> Your system has 2GB memory and it uses zswap with zsmalloc (which is
> good since it can allocate from the highmem zone) and zstd/lzo (which
> doesn't matter much). Somehow -- I couldn't figure out why -- it
> splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone:
>
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x0000000000000000-0x000000002fffffff]
> [    0.000000]   Normal   empty
> [    0.000000]   HighMem  [mem 0x0000000030000000-0x000000007fffffff]

That's really odd. But we are messing with "PowerMac3,6", so I don't 
really know what's right or wrong ...

--
Cheers,

David / dhildenb