Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value in rcu_process_callbacks

From: Paul E. McKenney
Date: Wed Nov 14 2018 - 11:05:49 EST

Next message: Strashko, Grygorii: "Re: [PATCH net-next 0/4] net: ethernet: ti: cpsw: allow vlan h/w timestamping"
Previous message: Jian-Hong Pan: "[PATCH V3 7/7] net: lorawan: List LORAWAN in menuconfig"
In reply to: Alexander Potapenko: "Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value in rcu_process_callbacks"
Next in thread: Kyungtae Kim: "Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value in rcu_process_callbacks"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, Nov 14, 2018 at 04:31:11PM +0100, Alexander Potapenko wrote:
> On Wed, Nov 14, 2018 at 4:09 PM Paul E. McKenney <paulmck@xxxxxxxxxxxxx> wrote:
> >
> > On Wed, Nov 14, 2018 at 04:03:33AM -0500, Kyungtae Kim wrote:
> > > We report two crashes in v4.19-rc8 (4.20-rc1 as well, I guess):
> > > (Unfortunately, there is no repro for those.)
> > >
> > > The two crashes seem to share the same issue.
> > > In both cases, (uninitialized) memory access violation occurs
> > > when "rdp->cblist" is about to be accessed (kernel/rcu/tree.c:2838,1728).
> > > I guess those are freed before the use, but I still haven't figured
> > > out the reason why.
> > > I'm looking forward to some help.
>
> First of all, I'd avoid reporting KMSAN bugs without clear reproducers.
> The tool is still in beta and may still give false positives due to
> either missed initialization or rare memory corruptions.

OK, I will set this aside, then, thank you!

Thanx, Paul

> > You lost me on this one. In both cases, rdp references a per-CPU
> > variable that is implicitly initialized to all zeroes, due to being
> > (sort of) a C-language global.
> >
> > If a callback is queued early, then the following lines in __call_rcu()
> > will make an honest list of that field because of the :
> >
> > if (rcu_segcblist_empty(&rdp->cblist))
> > rcu_segcblist_init(&rdp->cblist);
> >
> > Otherwise, when rcu_init() is invoked during early boot, we have this
> > in rcu_init_percpu_data(), which is called from rcutree_prepare_cpu()
> > which is called from rcu_init(), which is called from start_kernel():
> >
> > if (rcu_segcblist_empty(&rdp->cblist) && /* No early-boot CBs? */
> > !init_nocb_callback_list(rdp))
> > rcu_segcblist_init(&rdp->cblist); /* Re-enable callbacks. */
> >
> > So either init_nocb_callback_list() initializes the alternative callback
> > lists for a no-CBs CPU or rcu_segcblist_init() again makes an honest
> > list of that field.
> >
> > My guess is that your tool is missing the
> >
> > rdp = this_cpu_ptr(rsp->rda);
> >
> > in the __call_rcu() case, and also missing the
> >
> > struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
> >
> > Note that the ->rda field is explicitly compile-time initialized to
> > the base address of the per-CPU variable, which is rcu_preempt_data,
> > rcu_bh_data, or rcu_sched_data, depending on which RCU flavor is at hand.
> > (In v4.20-rc1, these are all merged into a single flavor to rule them all.)
> >
> > Alternatively, your tool might be missing the implicit initialization
> > of per-CPU variables.
> This used to be fine, but after rebasing to v4.20-rc2 I also started
> seeing strange reports on per-CPU variables. Taking a look.
> > Or maybe I am missing something. If so, please let me know what it is.
> >
> > Thanx, Paul
> >
> > > Crash log 1
> > > =========================================
> > > BUG: KMSAN: uninit-value in __rcu_process_callbacks
> > > kernel/rcu/tree.c:2838 [inline]
> > > BUG: KMSAN: uninit-value in rcu_process_callbacks+0x5ac/0x1cb0
> > > kernel/rcu/tree.c:2864
> > > CPU: 0 PID: 20 Comm: kauditd Not tainted 4.19.0-rc8+ #18
> > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > > Call Trace:
> > > <IRQ>
> > > __dump_stack lib/dump_stack.c:77 [inline]
> > > dump_stack+0x305/0x460 lib/dump_stack.c:113
> > > kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
> > > __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
> > > __rcu_process_callbacks kernel/rcu/tree.c:2838 [inline]
> > > rcu_process_callbacks+0x5ac/0x1cb0 kernel/rcu/tree.c:2864
> > > __do_softirq+0x5ff/0xa55 kernel/softirq.c:292
> > > invoke_softirq kernel/softirq.c:373 [inline]
> > > irq_exit+0x22d/0x270 kernel/softirq.c:414
> > > exiting_irq+0xe/0x10 arch/x86/include/asm/apic.h:536
> > > smp_apic_timer_interrupt+0x64/0x90 arch/x86/kernel/apic/apic.c:1059
> > > apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:869
> > > </IRQ>
> > > RIP: 0010:finish_lock_switch+0x2b/0x40 kernel/sched/core.c:2578
> > > Code: 48 89 e5 53 48 89 fb e8 e3 43 9a 00 8b b8 88 0c 00 00 48 8b 00
> > > 48 85 c0 75 12 48 89 df e8 7d 38 9a 00 c6 00 00 c6 03 00 fb 5b <5d> c3
> > > e8 de 42 9a 00 eb e7 66 66 66 2e 0f 1f 84 00 00 00 00 00 55
> > > RSP: 0018:ffff88010622fca0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
> > > RAX: ffff8801105bcc40 RBX: ffff8801061554c0 RCX: ffff8801105bdc40
> > > RDX: ffff8801105bdc40 RSI: aaaaaaaaaaaab000 RDI: ffffea00077ec560
> > > RBP: ffff88010622fca0 R08: ffffffff7fffffff R09: 0000000000000002
> > > R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800751cb880
> > > R13: 0000000000000000 R14: ffff880106155db8 R15: ffff88013fcb9c40
> > > finish_task_switch+0xe3/0x270 kernel/sched/core.c:2679
> > > context_switch kernel/sched/core.c:2832 [inline]
> > > __schedule+0x78f/0x8f0 kernel/sched/core.c:3479
> > > schedule+0x1cc/0x300 kernel/sched/core.c:3523
> > > kauditd_thread+0xc64/0xee0 kernel/audit.c:889
> > > kthread+0x5b1/0x5f0 kernel/kthread.c:247
> > > ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
> > >
> > > Uninit was created at:
> > > kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
> > > kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
> > > kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
> > > __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
> > > alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
> > > alloc_pages include/linux/gfp.h:511 [inline]
> > > alloc_slab_page mm/slub.c:1459 [inline]
> > > allocate_slab mm/slub.c:1604 [inline]
> > > new_slab+0x552/0x1f30 mm/slub.c:1675
> > > new_slab_objects mm/slub.c:2438 [inline]
> > > ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
> > > __slab_alloc mm/slub.c:2630 [inline]
> > > slab_alloc_node mm/slub.c:2693 [inline]
> > > slab_alloc mm/slub.c:2735 [inline]
> > > kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
> > > kmem_cache_zalloc include/linux/slab.h:697 [inline]
> > > avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
> > > avc_update_node+0x172/0x1ee0 security/selinux/avc.c:859
> > > avc_denied+0x312/0x360 security/selinux/avc.c:1024
> > > avc_has_perm_noaudit+0x733/0x770 security/selinux/avc.c:1155
> > > avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
> > > sock_has_perm security/selinux/hooks.c:4539 [inline]
> > > selinux_socket_sendmsg+0x297/0x360 security/selinux/hooks.c:4875
> > > security_socket_sendmsg+0x127/0x200 security/security.c:1410
> > > sock_sendmsg net/socket.c:628 [inline]
> > > ___sys_sendmsg+0xd5f/0x1290 net/socket.c:2116
> > > __sys_sendmsg net/socket.c:2154 [inline]
> > > __do_sys_sendmsg net/socket.c:2163 [inline]
> > > __se_sys_sendmsg+0x307/0x460 net/socket.c:2161
> > > __x64_sys_sendmsg+0x4a/0x70 net/socket.c:2161
> > > do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
> > > entry_SYSCALL_64_after_hwframe+0x63/0xe7
> > > =========================================================
> > >
> > > Crash log 2
> > > =========================================================
> > > BUG: KMSAN: uninit-value in rcu_accelerate_cbs+0x821/0x990
> > > kernel/rcu/tree.c:1728
> > > CPU: 0 PID: 10 Comm: rcu_sched Not tainted 4.19.0-rc8+ #18
> > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > > Call Trace:
> > > __dump_stack lib/dump_stack.c:77 [inline]
> > > dump_stack+0x305/0x460 lib/dump_stack.c:113
> > > kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
> > > __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
> > > rcu_accelerate_cbs+0x821/0x990 kernel/rcu/tree.c:1728
> > > __note_gp_changes+0x2ac/0x9e0 kernel/rcu/tree.c:1807
> > > rcu_gp_cleanup kernel/rcu/tree.c:2109 [inline]
> > > rcu_gp_kthread+0x3019/0x3990 kernel/rcu/tree.c:2236
> > > kthread+0x5b1/0x5f0 kernel/kthread.c:247
> > > ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
> > >
> > > Uninit was created at:
> > > kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
> > > kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
> > > kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
> > > __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
> > > alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
> > > alloc_pages include/linux/gfp.h:511 [inline]
> > > alloc_slab_page mm/slub.c:1459 [inline]
> > > allocate_slab mm/slub.c:1604 [inline]
> > > new_slab+0x552/0x1f30 mm/slub.c:1675
> > > new_slab_objects mm/slub.c:2438 [inline]
> > > ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
> > > __slab_alloc mm/slub.c:2630 [inline]
> > > slab_alloc_node mm/slub.c:2693 [inline]
> > > slab_alloc mm/slub.c:2735 [inline]
> > > kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
> > > kmem_cache_zalloc include/linux/slab.h:697 [inline]
> > > avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
> > > avc_insert security/selinux/avc.c:696 [inline]
> > > avc_compute_av+0x31e/0x1050 security/selinux/avc.c:1008
> > > avc_has_perm_noaudit+0x516/0x770 security/selinux/avc.c:1149
> > > avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
> > > selinux_socket_create+0x248/0x3c0 security/selinux/hooks.c:4560
> > > security_socket_create+0x146/0x210 security/security.c:1372
> > > __sock_create+0x26b/0xf30 net/socket.c:1232
> > > sock_create net/socket.c:1317 [inline]
> > > __sys_socket+0x180/0x670 net/socket.c:1347
> > > __do_sys_socket net/socket.c:1356 [inline]
> > > __se_sys_socket+0x8d/0xb0 net/socket.c:1354
> > > __x64_sys_socket+0x4a/0x70 net/socket.c:1354
> > > do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
> > > entry_SYSCALL_64_after_hwframe+0x63/0xe7
> > > =========================================================
> > >
> > > Thanks,
> > > Kyungtae Kim
> > >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "syzkaller" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+unsubscribe@xxxxxxxxxxxxxxxxx
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Alexander Potapenko
> Software Engineer
>
> Google Germany GmbH
> Erika-Mann-Straße, 33
> 80636 München
>
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> Registergericht und -nummer: Hamburg, HRB 86891
> Sitz der Gesellschaft: Hamburg
>

Next message: Strashko, Grygorii: "Re: [PATCH net-next 0/4] net: ethernet: ti: cpsw: allow vlan h/w timestamping"
Previous message: Jian-Hong Pan: "[PATCH V3 7/7] net: lorawan: List LORAWAN in menuconfig"
In reply to: Alexander Potapenko: "Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value in rcu_process_callbacks"
Next in thread: Kyungtae Kim: "Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value in rcu_process_callbacks"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]