Re: INFO: rcu detected stall in kvm_mmu_unprotect_page

[Dmitry Vyukov]

On Mon, Sep 10, 2018 at 9:41 AM, syzbot
<syzbot+3656615bd54615add56a@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit:    f8f65382c98a Merge tag 'for-linus' of git://git.kernel.org..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=12d31571400000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=8f59875069d721b6
> dashboard link: https://syzkaller.appspot.com/bug?extid=3656615bd54615add56a
> compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=124a5346400000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+3656615bd54615add56a@xxxxxxxxxxxxxxxxxxxxxxxxx

#syz dup: INFO: rcu detected stall in kvm_vcpu_ioctl

> 8021q: adding VLAN 0 to HW filter on device team0
> 8021q: adding VLAN 0 to HW filter on device team0
> 8021q: adding VLAN 0 to HW filter on device team0
> hrtimer: interrupt took 30378 ns
> L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and
> https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html for details.
> rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> rcu:    (detected by 1, t=10502 jiffies, g=7113, q=509)
> rcu: All QSes seen, last rcu_preempt kthread activity 10498
> (4295042895-4295032397), jiffies_till_next_fqs=1, root ->qsmask 0x0
> syz-executor0   R  running task    22808  7363   5346 0x00000000
> Call Trace:
>  <IRQ>
>  sched_show_task.cold.83+0x2b6/0x30a kernel/sched/core.c:5296
>  print_other_cpu_stall.cold.79+0xa83/0xba5 kernel/rcu/tree.c:1430
>  check_cpu_stall kernel/rcu/tree.c:1557 [inline]
>  __rcu_pending kernel/rcu/tree.c:3276 [inline]
>  rcu_pending kernel/rcu/tree.c:3319 [inline]
>  rcu_check_callbacks+0xafc/0x1990 kernel/rcu/tree.c:2665
>  update_process_times+0x2d/0x70 kernel/time/timer.c:1636
>  tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164
>  tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274
>  __run_hrtimer kernel/time/hrtimer.c:1398 [inline]
>  __hrtimer_run_queues+0x41c/0x10d0 kernel/time/hrtimer.c:1460
>  hrtimer_interrupt+0x313/0x780 kernel/time/hrtimer.c:1518
>  local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1029 [inline]
>  smp_apic_timer_interrupt+0x1a1/0x760 arch/x86/kernel/apic/apic.c:1054
>  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:864
>  </IRQ>
> RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:788
> [inline]
> RIP: 0010:lock_acquire+0x268/0x520 kernel/locking/lockdep.c:3904
> Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 44 02 00 00 48 83 3d af
> 39 f1 07 00 0f 84 c3 01 00 00 48 8b bd 20 ff ff ff 57 9d <0f> 1f 44 00 00 48
> b8 00 00 00 00 00 fc ff df 48 01 c3 48 c7 03 00
> RSP: 0018:ffff8801c664eb20 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
> RAX: dffffc0000000000 RBX: 1ffff10038cc9d69 RCX: 0000000000000000
> RDX: 1ffffffff12a4535 RSI: 0000000000000000 RDI: 0000000000000282
> RBP: ffff8801c664ec10 R08: ffff8801c10e6e70 R09: 0000000000000002
> R10: ffff8801c10e6e50 R11: f333abdc872bc457 R12: ffff8801c10e6580
> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>  _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144
>  spin_lock include/linux/spinlock.h:329 [inline]
>  kvm_mmu_unprotect_page+0xf9/0x450 arch/x86/kvm/mmu.c:2720
>  retry_instruction arch/x86/kvm/x86.c:5982 [inline]
>  x86_emulate_instruction+0x635/0x1fc0 arch/x86/kvm/x86.c:6210
>  kvm_mmu_page_fault+0x396/0x1b30 arch/x86/kvm/mmu.c:5280
>  handle_ept_violation+0x29e/0x6a0 arch/x86/kvm/vmx.c:7679
>  vmx_handle_exit+0x2f7/0x17e0 arch/x86/kvm/vmx.c:10115
>  vcpu_enter_guest+0x14a9/0x62e0 arch/x86/kvm/x86.c:7648
>  vcpu_run arch/x86/kvm/x86.c:7711 [inline]
>  kvm_arch_vcpu_ioctl_run+0x375/0x16e0 arch/x86/kvm/x86.c:7888
>  kvm_vcpu_ioctl+0x72b/0x1150 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2590
>  vfs_ioctl fs/ioctl.c:46 [inline]
>  file_ioctl fs/ioctl.c:501 [inline]
>  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
>  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
>  __do_sys_ioctl fs/ioctl.c:709 [inline]
>  __se_sys_ioctl fs/ioctl.c:707 [inline]
>  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
>  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
>  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> RIP: 0033:0x457099
> Code: fd b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
> 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
> 0f 83 cb b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00
> RSP: 002b:00007fff8bfb2748 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 000000000212b914 RCX: 0000000000457099
> RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000006
> RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
> R13: 00000000004cf730 R14: 00000000004c59b9 R15: 0000000000000000
> rcu: rcu_preempt kthread starved for 10498 jiffies! g7113 f0x2
> RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
> rcu: RCU grace-period kthread stack dump:
> rcu_preempt     R  running task    22872    10      2 0x80000000
> Call Trace:
>  context_switch kernel/sched/core.c:2825 [inline]
>  __schedule+0x86c/0x1ed0 kernel/sched/core.c:3473
>  schedule+0xfe/0x460 kernel/sched/core.c:3517
>  schedule_timeout+0x140/0x260 kernel/time/timer.c:1804
>  rcu_gp_kthread+0x9d9/0x2310 kernel/rcu/tree.c:2194
>  kthread+0x35a/0x420 kernel/kthread.c:246
>  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:413
> rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> rcu:    (detected by 1, t=10652 jiffies, g=-1171, q=0)
> rcu: All QSes seen, last rcu_sched kthread activity 10649
> (4295043046-4295032397), jiffies_till_next_fqs=1, root ->qsmask 0x0
> syz-executor0   R  running task    22808  7363   5346 0x80000000
> Call Trace:
>  <IRQ>
>  sched_show_task.cold.83+0x2b6/0x30a kernel/sched/core.c:5296
>  print_other_cpu_stall.cold.79+0xa83/0xba5 kernel/rcu/tree.c:1430
>  check_cpu_stall kernel/rcu/tree.c:1557 [inline]
>  __rcu_pending kernel/rcu/tree.c:3276 [inline]
>  rcu_pending kernel/rcu/tree.c:3319 [inline]
>  rcu_check_callbacks+0xafc/0x1990 kernel/rcu/tree.c:2665
>  update_process_times+0x2d/0x70 kernel/time/timer.c:1636
>  tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164
>  tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274
>  __run_hrtimer kernel/time/hrtimer.c:1398 [inline]
>  __hrtimer_run_queues+0x41c/0x10d0 kernel/time/hrtimer.c:1460
>  hrtimer_interrupt+0x313/0x780 kernel/time/hrtimer.c:1518
>  local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1029 [inline]
>  smp_apic_timer_interrupt+0x1a1/0x760 arch/x86/kernel/apic/apic.c:1054
>  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:864
>  </IRQ>
> RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:788
> [inline]
> RIP: 0010:lock_acquire+0x268/0x520 kernel/locking/lockdep.c:3904
> Code: 00 00 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 44 02 00 00 48 83 3d af
> 39 f1 07 00 0f 84 c3 01 00 00 48 8b bd 20 ff ff ff 57 9d <0f> 1f 44 00 00 48
> b8 00 00 00 00 00 fc ff df 48 01 c3 48 c7 03 00
> RSP: 0018:ffff8801c664eb20 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
> RAX: dffffc0000000000 RBX: 1ffff10038cc9d69 RCX: 0000000000000000
> RDX: 1ffffffff12a4535 RSI: 0000000000000000 RDI: 0000000000000282
> RBP: ffff8801c664ec10 R08: ffff8801c10e6e70 R09: 0000000000000002
> R10: ffff8801c10e6e50 R11: f333abdc872bc457 R12: ffff8801c10e6580
> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
>  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>  _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144
>  spin_lock include/linux/spinlock.h:329 [inline]
>  kvm_mmu_unprotect_page+0xf9/0x450 arch/x86/kvm/mmu.c:2720
>  retry_instruction arch/x86/kvm/x86.c:5982 [inline]
>  x86_emulate_instruction+0x635/0x1fc0 arch/x86/kvm/x86.c:6210
>  kvm_mmu_page_fault+0x396/0x1b30 arch/x86/kvm/mmu.c:5280
>  handle_ept_violation+0x29e/0x6a0 arch/x86/kvm/vmx.c:7679
>  vmx_handle_exit+0x2f7/0x17e0 arch/x86/kvm/vmx.c:10115
>  vcpu_enter_guest+0x14a9/0x62e0 arch/x86/kvm/x86.c:7648
>  vcpu_run arch/x86/kvm/x86.c:7711 [inline]
>  kvm_arch_vcpu_ioctl_run+0x375/0x16e0 arch/x86/kvm/x86.c:7888
>  kvm_vcpu_ioctl+0x72b/0x1150 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2590
>  vfs_ioctl fs/ioctl.c:46 [inline]
>  file_ioctl fs/ioctl.c:501 [inline]
>  do_vfs_ioctl+0x1de/0x1720 fs/ioctl.c:685
>  ksys_ioctl+0xa9/0xd0 fs/ioctl.c:702
>  __do_sys_ioctl fs/ioctl.c:709 [inline]
>  __se_sys_ioctl fs/ioctl.c:707 [inline]
>  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:707
>  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
>  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> RIP: 0033:0x457099
> Code: fd b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
> 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
> 0f 83 cb b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00
> RSP: 002b:00007fff8bfb2748 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 000000000212b914 RCX: 0000000000457099
> RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000006
> RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
> R13: 00000000004cf730 R14: 00000000004c59b9 R15: 0000000000000000
> rcu: rcu_sched kthread starved for 10649 jiffies! g-1171 f0x2
> RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
> rcu: RCU grace-period kthread stack dump:
> rcu_sched       R  running task    24600    11      2 0x80000000
> Call Trace:
>  context_switch kernel/sched/core.c:2825 [inline]
>  __schedule+0x86c/0x1ed0 kernel/sched/core.c:3473
>  schedule+0xfe/0x460 kernel/sched/core.c:3517
>  schedule_timeout+0x140/0x260 kernel/time/timer.c:1804
>  rcu_gp_kthread+0x9d9/0x2310 kernel/rcu/tree.c:2194
>  kthread+0x35a/0x420 kernel/kthread.c:246
>  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:413
>
>
> ---
> This bug is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@xxxxxxxxxxxxxxxxx
>
> syzbot will keep track of this bug report. See:
> https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
> syzbot.
> syzbot can test patches for this bug, for details see:
> https://goo.gl/tpsmEJ#testing-patches
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bugs+unsubscribe@xxxxxxxxxxxxxxxxx
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000cf740005757f7a21%40google.com.
> For more options, visit https://groups.google.com/d/optout.