Re: [PATCH] mm/slub: fix a deadlock in shuffle_freelist()

From: Qian Cai
Date: Mon Sep 16 2019 - 17:31:48 EST

Next message: Kees Cook: "[PATCH] usercopy: Skip HIGHMEM page checking"
Previous message: Linus Torvalds: "Re: [RFC] Improve memset"
In reply to: Sebastian Andrzej Siewior: "Re: [PATCH] mm/slub: fix a deadlock in shuffle_freelist()"
Next in thread: Sebastian Andrzej Siewior: "Re: [PATCH] mm/slub: fix a deadlock in shuffle_freelist()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Mon, 2019-09-16 at 21:51 +0200, Sebastian Andrzej Siewior wrote:
> On 2019-09-16 10:01:27 [-0400], Qian Cai wrote:
> > On Mon, 2019-09-16 at 11:03 +0200, Sebastian Andrzej Siewior wrote:
> > > On 2019-09-13 12:27:44 [-0400], Qian Cai wrote:
> > > â
> > > > Chain exists of:
> > > > random_write_wait.lock --> &rq->lock --> batched_entropy_u32.lock
> > > >
> > > > Possible unsafe locking scenario:
> > > >
> > > > CPU0 CPU1
> > > > ---- ----
> > > > lock(batched_entropy_u32.lock);
> > > > lock(&rq->lock);
> > > > lock(batched_entropy_u32.lock);
> > > > lock(random_write_wait.lock);
> > >
> > > would this deadlock still occur if lockdep knew that
> > > batched_entropy_u32.lock on CPU0 could be acquired at the same time
> > > as CPU1 acquired its batched_entropy_u32.lock?
> >
> > I suppose that might fix it too if it can teach the lockdep the trick, but it
> > would be better if there is a patch if you have something in mind that could be
> > tested to make sure.
>
> get_random_bytes() is heavier than get_random_int() so I would prefer to
> avoid its usage to fix what looks like a false positive report from
> lockdep.
> But no, I don't have a patch sitting around. A lock in per-CPU memory
> could lead to the scenario mentioned above if the lock could be obtained
> cross-CPU it just isn't so in that case. So I don't think it is that
> simple.

get_random_u64() is also busted.

[ÂÂ752.925079] WARNING: possible circular locking dependency detected
[ÂÂ752.931951] 5.3.0-rc8-next-20190915+ #2 Tainted: GÂÂÂÂÂÂÂÂÂÂÂÂÂLÂÂÂ
[ÂÂ752.938906] ------------------------------------------------------
[ÂÂ752.945774] ls/9665 is trying to acquire lock:
[ÂÂ752.950905] ffff90001311fef8 (random_write_wait.lock){..-.}, at:
__wake_up_common_lock+0xa8/0x11c
[ÂÂ752.960481]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂbut task is already holding lock:
[ÂÂ752.967698] ffff008abc7b9c00 (batched_entropy_u64.lock){....}, at:
get_random_u64+0x6c/0x1dc
[ÂÂ752.976835]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂwhich lock already depends on the new lock.

[ÂÂ752.987089]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂthe existing dependency chain (in reverse order) is:
[ÂÂ752.995953]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-> #4 (batched_entropy_u64.lock){....}:
[ÂÂ753.003702]ÂÂÂÂÂÂÂÂlock_acquire+0x320/0x364
[ÂÂ753.008577]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x7c/0x9c
[ÂÂ753.014145]ÂÂÂÂÂÂÂÂget_random_u64+0x6c/0x1dc
[ÂÂ753.019109]ÂÂÂÂÂÂÂÂadd_to_free_area_random+0x54/0x1c8
[ÂÂ753.024851]ÂÂÂÂÂÂÂÂfree_one_page+0x86c/0xc28
[ÂÂ753.029818]ÂÂÂÂÂÂÂÂ__free_pages_ok+0x69c/0xdac
[ÂÂ753.034960]ÂÂÂÂÂÂÂÂ__free_pages+0xbc/0xf8
[ÂÂ753.039663]ÂÂÂÂÂÂÂÂ__free_pages_core+0x2ac/0x3c0
[ÂÂ753.044973]ÂÂÂÂÂÂÂÂmemblock_free_pages+0xe0/0xf8
[ÂÂ753.050281]ÂÂÂÂÂÂÂÂ__free_pages_memory+0xcc/0xfc
[ÂÂ753.055588]ÂÂÂÂÂÂÂÂ__free_memory_core+0x70/0x78
[ÂÂ753.060809]ÂÂÂÂÂÂÂÂfree_low_memory_core_early+0x148/0x18c
[ÂÂ753.066897]ÂÂÂÂÂÂÂÂmemblock_free_all+0x18/0x54
[ÂÂ753.072033]ÂÂÂÂÂÂÂÂmem_init+0x9c/0x160
[ÂÂ753.076472]ÂÂÂÂÂÂÂÂmm_init+0x14/0x38
[ÂÂ753.080737]ÂÂÂÂÂÂÂÂstart_kernel+0x19c/0x52c
[ÂÂ753.085607]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-> #3 (&(&zone->lock)->rlock){..-.}:
[ÂÂ753.093092]ÂÂÂÂÂÂÂÂlock_acquire+0x320/0x364
[ÂÂ753.097964]ÂÂÂÂÂÂÂÂ_raw_spin_lock+0x64/0x80
[ÂÂ753.102839]ÂÂÂÂÂÂÂÂrmqueue_bulk+0x50/0x15a0
[ÂÂ753.107712]ÂÂÂÂÂÂÂÂget_page_from_freelist+0x2260/0x29dc
[ÂÂ753.113627]ÂÂÂÂÂÂÂÂ__alloc_pages_nodemask+0x36c/0x1ce0
[ÂÂ753.119457]ÂÂÂÂÂÂÂÂalloc_page_interleave+0x34/0x17c
[ÂÂ753.125023]ÂÂÂÂÂÂÂÂalloc_pages_current+0x80/0xe0
[ÂÂ753.130334]ÂÂÂÂÂÂÂÂallocate_slab+0xfc/0x1d80
[ÂÂ753.135296]ÂÂÂÂÂÂÂÂ___slab_alloc+0x5d4/0xa70
[ÂÂ753.140257]ÂÂÂÂÂÂÂÂkmem_cache_alloc+0x588/0x66c
[ÂÂ753.145480]ÂÂÂÂÂÂÂÂ__debug_object_init+0x9d8/0xbac
[ÂÂ753.150962]ÂÂÂÂÂÂÂÂdebug_object_init+0x40/0x50
[ÂÂ753.156098]ÂÂÂÂÂÂÂÂhrtimer_init+0x38/0x2b4
[ÂÂ753.160885]ÂÂÂÂÂÂÂÂinit_dl_task_timer+0x24/0x44
[ÂÂ753.166108]ÂÂÂÂÂÂÂÂ__sched_fork+0xc0/0x168
[ÂÂ753.170894]ÂÂÂÂÂÂÂÂinit_idle+0x80/0x3d8
[ÂÂ753.175420]ÂÂÂÂÂÂÂÂidle_thread_get+0x60/0x8c
[ÂÂ753.180385]ÂÂÂÂÂÂÂÂ_cpu_up+0x10c/0x348
[ÂÂ753.184824]ÂÂÂÂÂÂÂÂdo_cpu_up+0x114/0x170
[ÂÂ753.189437]ÂÂÂÂÂÂÂÂcpu_up+0x20/0x2c
[ÂÂ753.193615]ÂÂÂÂÂÂÂÂsmp_init+0xf8/0x1bc
[ÂÂ753.198054]ÂÂÂÂÂÂÂÂkernel_init_freeable+0x198/0x26c
[ÂÂ753.203622]ÂÂÂÂÂÂÂÂkernel_init+0x18/0x334
[ÂÂ753.208323]ÂÂÂÂÂÂÂÂret_from_fork+0x10/0x18
[ÂÂ753.213107]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-> #2 (&rq->lock){-.-.}:
[ÂÂ753.219550]ÂÂÂÂÂÂÂÂlock_acquire+0x320/0x364
[ÂÂ753.224423]ÂÂÂÂÂÂÂÂ_raw_spin_lock+0x64/0x80
[ÂÂ753.229299]ÂÂÂÂÂÂÂÂtask_fork_fair+0x64/0x22c
[ÂÂ753.234261]ÂÂÂÂÂÂÂÂsched_fork+0x24c/0x3d8
[ÂÂ753.238962]ÂÂÂÂÂÂÂÂcopy_process+0xa60/0x29b0
[ÂÂ753.243921]ÂÂÂÂÂÂÂÂ_do_fork+0xb8/0xa64
[ÂÂ753.248360]ÂÂÂÂÂÂÂÂkernel_thread+0xc4/0xf4
[ÂÂ753.253147]ÂÂÂÂÂÂÂÂrest_init+0x30/0x320
[ÂÂ753.257673]ÂÂÂÂÂÂÂÂarch_call_rest_init+0x10/0x18
[ÂÂ753.262980]ÂÂÂÂÂÂÂÂstart_kernel+0x424/0x52c
[ÂÂ753.267849]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-> #1 (&p->pi_lock){-.-.}:
[ÂÂ753.274467]ÂÂÂÂÂÂÂÂlock_acquire+0x320/0x364
[ÂÂ753.279342]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x7c/0x9c
[ÂÂ753.284910]ÂÂÂÂÂÂÂÂtry_to_wake_up+0x74/0x128c
[ÂÂ753.289959]ÂÂÂÂÂÂÂÂdefault_wake_function+0x38/0x48
[ÂÂ753.295440]ÂÂÂÂÂÂÂÂpollwake+0x118/0x158
[ÂÂ753.299967]ÂÂÂÂÂÂÂÂ__wake_up_common+0x16c/0x240
[ÂÂ753.305187]ÂÂÂÂÂÂÂÂ__wake_up_common_lock+0xc8/0x11c
[ÂÂ753.310754]ÂÂÂÂÂÂÂÂ__wake_up+0x3c/0x4c
[ÂÂ753.315193]ÂÂÂÂÂÂÂÂaccount+0x390/0x3e0
[ÂÂ753.319632]ÂÂÂÂÂÂÂÂextract_entropy+0x2cc/0x37c
[ÂÂ753.324766]ÂÂÂÂÂÂÂÂ_xfer_secondary_pool+0x35c/0x3c4
[ÂÂ753.330333]ÂÂÂÂÂÂÂÂpush_to_pool+0x54/0x308
[ÂÂ753.335119]ÂÂÂÂÂÂÂÂprocess_one_work+0x558/0xb1c
[ÂÂ753.340339]ÂÂÂÂÂÂÂÂworker_thread+0x494/0x650
[ÂÂ753.345300]ÂÂÂÂÂÂÂÂkthread+0x1cc/0x1e8
[ÂÂ753.349739]ÂÂÂÂÂÂÂÂret_from_fork+0x10/0x18
[ÂÂ753.354522]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-> #0 (random_write_wait.lock){..-.}:
[ÂÂ753.362093]ÂÂÂÂÂÂÂÂvalidate_chain+0xfcc/0x2fd4
[ÂÂ753.367227]ÂÂÂÂÂÂÂÂ__lock_acquire+0x868/0xc2c
[ÂÂ753.372274]ÂÂÂÂÂÂÂÂlock_acquire+0x320/0x364
[ÂÂ753.377147]ÂÂÂÂÂÂÂÂ_raw_spin_lock_irqsave+0x7c/0x9c
[ÂÂ753.382715]ÂÂÂÂÂÂÂÂ__wake_up_common_lock+0xa8/0x11c
[ÂÂ753.388282]ÂÂÂÂÂÂÂÂ__wake_up+0x3c/0x4c
[ÂÂ753.392720]ÂÂÂÂÂÂÂÂaccount+0x390/0x3e0
[ÂÂ753.397159]ÂÂÂÂÂÂÂÂextract_entropy+0x2cc/0x37c
[ÂÂ753.402292]ÂÂÂÂÂÂÂÂcrng_reseed+0x60/0x350
[ÂÂ753.406991]ÂÂÂÂÂÂÂÂ_extract_crng+0xd8/0x164
[ÂÂ753.411864]ÂÂÂÂÂÂÂÂcrng_reseed+0x7c/0x350
[ÂÂ753.416563]ÂÂÂÂÂÂÂÂ_extract_crng+0xd8/0x164
[ÂÂ753.421436]ÂÂÂÂÂÂÂÂget_random_u64+0xec/0x1dc
[ÂÂ753.426396]ÂÂÂÂÂÂÂÂarch_mmap_rnd+0x18/0x78
[ÂÂ753.431187]ÂÂÂÂÂÂÂÂload_elf_binary+0x6d0/0x1730
[ÂÂ753.436411]ÂÂÂÂÂÂÂÂsearch_binary_handler+0x10c/0x35c
[ÂÂ753.442067]ÂÂÂÂÂÂÂÂ__do_execve_file+0xb58/0xf7c
[ÂÂ753.447287]ÂÂÂÂÂÂÂÂ__arm64_sys_execve+0x6c/0xa4
[ÂÂ753.452509]ÂÂÂÂÂÂÂÂel0_svc_handler+0x170/0x240
[ÂÂ753.457643]ÂÂÂÂÂÂÂÂel0_svc+0x8/0xc
[ÂÂ753.461732]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂother info that might help us debug this:

[ÂÂ753.471812] Chain exists of:
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂrandom_write_wait.lock --> &(&zone->lock)->rlock -->
batched_entropy_u64.lock

[ÂÂ753.486588]ÂÂPossible unsafe locking scenario:

[ÂÂ753.493890]ÂÂÂÂÂÂÂÂCPU0ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂCPU1
[ÂÂ753.499108]ÂÂÂÂÂÂÂÂ----ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ----
[ÂÂ753.504324]ÂÂÂlock(batched_entropy_u64.lock);
[ÂÂ753.509372]ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlock(&(&zone->lock)->rlock);
[ÂÂ753.516675]ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂlock(batched_entropy_u64.lock);
[ÂÂ753.524238]ÂÂÂlock(random_write_wait.lock);
[ÂÂ753.529113]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ*** DEADLOCK ***

[ÂÂ753.537111] 1 lock held by ls/9665:
[ÂÂ753.541287]ÂÂ#0: ffff008abc7b9c00 (batched_entropy_u64.lock){....}, at:
get_random_u64+0x6c/0x1dc
[ÂÂ753.550858]Â
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂstack backtrace:
[ÂÂ753.556602] CPU: 121 PID: 9665 Comm: ls Tainted: GÂÂÂÂÂÂÂÂÂÂÂÂÂLÂÂÂÂ5.3.0-
rc8-next-20190915+ #2
[ÂÂ753.565987] Hardware name: HPE Apollo 70ÂÂÂÂÂÂÂÂÂÂÂÂÂ/C01_APACHE_MBÂÂÂÂÂÂÂÂÂ,
BIOS L50_5.13_1.11 06/18/2019
[ÂÂ753.576414] Call trace:
[ÂÂ753.579553]ÂÂdump_backtrace+0x0/0x264
[ÂÂ753.583905]ÂÂshow_stack+0x20/0x2c
[ÂÂ753.587911]ÂÂdump_stack+0xd0/0x140
[ÂÂ753.592003]ÂÂprint_circular_bug+0x368/0x380
[ÂÂ753.596876]ÂÂcheck_noncircular+0x28c/0x294
[ÂÂ753.601664]ÂÂvalidate_chain+0xfcc/0x2fd4
[ÂÂ753.606276]ÂÂ__lock_acquire+0x868/0xc2c
[ÂÂ753.610802]ÂÂlock_acquire+0x320/0x364
[ÂÂ753.615154]ÂÂ_raw_spin_lock_irqsave+0x7c/0x9c
[ÂÂ753.620202]ÂÂ__wake_up_common_lock+0xa8/0x11c
[ÂÂ753.625248]ÂÂ__wake_up+0x3c/0x4c
[ÂÂ753.629171]ÂÂaccount+0x390/0x3e0
[ÂÂ753.633095]ÂÂextract_entropy+0x2cc/0x37c
[ÂÂ753.637708]ÂÂcrng_reseed+0x60/0x350
[ÂÂ753.641887]ÂÂ_extract_crng+0xd8/0x164
[ÂÂ753.646238]ÂÂcrng_reseed+0x7c/0x350
[ÂÂ753.650417]ÂÂ_extract_crng+0xd8/0x164
[ÂÂ753.654768]ÂÂget_random_u64+0xec/0x1dc
[ÂÂ753.659208]ÂÂarch_mmap_rnd+0x18/0x78
[ÂÂ753.663474]ÂÂload_elf_binary+0x6d0/0x1730
[ÂÂ753.668173]ÂÂsearch_binary_handler+0x10c/0x35c
[ÂÂ753.673308]ÂÂ__do_execve_file+0xb58/0xf7c
[ÂÂ753.678007]ÂÂ__arm64_sys_execve+0x6c/0xa4
[ÂÂ753.682707]ÂÂel0_svc_handler+0x170/0x240
[ÂÂ753.687319]ÂÂel0_svc+0x8/0xc

Next message: Kees Cook: "[PATCH] usercopy: Skip HIGHMEM page checking"
Previous message: Linus Torvalds: "Re: [RFC] Improve memset"
In reply to: Sebastian Andrzej Siewior: "Re: [PATCH] mm/slub: fix a deadlock in shuffle_freelist()"
Next in thread: Sebastian Andrzej Siewior: "Re: [PATCH] mm/slub: fix a deadlock in shuffle_freelist()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]