Re: kvm: WARNING in x86_emulate_insn

[Dmitry Vyukov]

On Tue, Jan 17, 2017 at 12:34 PM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
> On Fri, Jan 13, 2017 at 6:47 PM, Radim KrÄmÃÅ <rkrcmar@xxxxxxxxxx> wrote:
>> 2017-01-12 14:55+0100, Dmitry Vyukov:
>>> Hello,
>>>
>>> I've got the following WARNING in x86_emulate_insn while running
>>> syzkaller fuzzer:
>>>
>>> WARNING: CPU: 2 PID: 18646 at arch/x86/kvm/emulate.c:5558
>>> x86_emulate_insn+0x16a5/0x4090 arch/x86/kvm/emulate.c:5572
>>> Modules linked in:
>>> CPU: 2 PID: 18646 Comm: syz-executor Not tainted 4.10.0-rc3+ #155
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>> Call Trace:
>>>  __dump_stack lib/dump_stack.c:15 [inline]
>>>  dump_stack+0x292/0x3a2 lib/dump_stack.c:51
>>>  __warn+0x19f/0x1e0 kernel/panic.c:547
>>>  warn_slowpath_null+0x2c/0x40 kernel/panic.c:582
>>>  x86_emulate_insn+0x16a5/0x4090 arch/x86/kvm/emulate.c:5572
>>>  x86_emulate_instruction+0x403/0x1cc0 arch/x86/kvm/x86.c:5618
>>>  emulate_instruction arch/x86/include/asm/kvm_host.h:1127 [inline]
>>>  handle_exception+0x594/0xfd0 arch/x86/kvm/vmx.c:5762
>>>  vmx_handle_exit+0x2b7/0x38b0 arch/x86/kvm/vmx.c:8625
>>>  vcpu_enter_guest arch/x86/kvm/x86.c:6888 [inline]
>>>  vcpu_run arch/x86/kvm/x86.c:6947 [inline]
>>>  kvm_arch_vcpu_ioctl_run+0xf3d/0x4660 arch/x86/kvm/x86.c:7105
>>>  kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
>>>  vfs_ioctl fs/ioctl.c:43 [inline]
>>>  do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
>>>  SYSC_ioctl fs/ioctl.c:698 [inline]
>>>  SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
>>>  entry_SYSCALL_64_fastpath+0x1f/0xc2
>>> RIP: 0033:0x445329
>>> RSP: 002b:00007f9e6e22fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
>>> RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 0000000000445329
>>> RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000018
>>> RBP: 00000000006deb40 R08: 0000000000000000 R09: 0000000000000000
>>> R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700150
>>> R13: 0000000000000000 R14: 00007f9e6e2309c0 R15: 00007f9e6e230700
>>> ---[ end trace 6b54f749506b620c ]---
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 2 PID: 18646 at arch/x86/kvm/x86.c:366
>>> exception_type+0x73/0x80 arch/x86/kvm/x86.c:366
>>> Modules linked in:
>>> CPU: 2 PID: 18646 Comm: syz-executor Tainted: G        W       4.10.0-rc3+ #155
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>> Call Trace:
>>>  __dump_stack lib/dump_stack.c:15 [inline]
>>>  dump_stack+0x292/0x3a2 lib/dump_stack.c:51
>>>  __warn+0x19f/0x1e0 kernel/panic.c:547
>>>  warn_slowpath_null+0x2c/0x40 kernel/panic.c:582
>>>  exception_type+0x73/0x80 arch/x86/kvm/x86.c:366
>>>  x86_emulate_instruction+0x1356/0x1cc0 arch/x86/kvm/x86.c:5664
>>>  emulate_instruction arch/x86/include/asm/kvm_host.h:1127 [inline]
>>>  handle_exception+0x594/0xfd0 arch/x86/kvm/vmx.c:5762
>>>  vmx_handle_exit+0x2b7/0x38b0 arch/x86/kvm/vmx.c:8625
>>>  vcpu_enter_guest arch/x86/kvm/x86.c:6888 [inline]
>>>  vcpu_run arch/x86/kvm/x86.c:6947 [inline]
>>>  kvm_arch_vcpu_ioctl_run+0xf3d/0x4660 arch/x86/kvm/x86.c:7105
>>>  kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
>>>  vfs_ioctl fs/ioctl.c:43 [inline]
>>>  do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
>>>  SYSC_ioctl fs/ioctl.c:698 [inline]
>>>  SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
>>>  entry_SYSCALL_64_fastpath+0x1f/0xc2
>>> RIP: 0033:0x445329
>>> RSP: 002b:00007f9e6e22fb58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
>>> RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 0000000000445329
>>> RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000018
>>> RBP: 00000000006deb40 R08: 0000000000000000 R09: 0000000000000000
>>> R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700150
>>> R13: 0000000000000000 R14: 00007f9e6e2309c0 R15: 00007f9e6e230700
>>> ---[ end trace 6b54f749506b620d ]---
>>>
>>> On commit ba836a6f5ab1243ff5e08a941a2d1de8b31244e1.
>>>
>>> Unfortunately I can't reproduce it with a C program.
>>> It reproduces with the following syzkaller program within a minute, though:
>>> https://gist.githubusercontent.com/dvyukov/d09118fb9d986a9385487d80a1b50680/raw/884c68d22c3a80778ae596a6c5daf7467ea41b68/gistfile1.txt
>>> It can be executed following these instructions:
>>> https://github.com/google/syzkaller/wiki/How-to-execute-syzkaller-programs
>>> I run syz-execprog as:
>>> ./syz-execprog -repeat=0 -procs=8 -sandbox=none gistfile1.txt
>>>
>>> Note that syz_kvm_setup_cpu is a pseudo syscall that setups vcpu into
>>> a complex state:
>>> https://github.com/google/syzkaller/blob/master/executor/common_kvm_amd64.h#L271
>>>
>>> My bet would be on some race where VM memory is overwritten
>>> concurrently, and it affects either guest execution or
>>> emulate_instruction in a bad way...
>>
>> Yeah, all functions that return X86EMUL_PROPAGATE_FAULT seem to set
>> exception.vector to something sane.  The only easy way to get a bad value there
>> is when x86_emulate_instruction() clears it to -1U, but I don't see how a race
>> would play out.
>>
>> Anyway, I can't reproduce on bare metal [got another warning, see below].
>> Will try after rebuilding a guest kernel.
>
>
> I've dumped exception before the warning and they all look the same:
>
> [  211.608578] vector=255 error_code_valid=0 error_code=0
> nested_page_fault=0 address=          (null)
> [  211.609793] ------------[ cut here ]------------
> [  211.612438] WARNING: CPU: 3 PID: 1876 at
> arch/x86/kvm/emulate.c:5611 x86_emulate_insn+0x17bf/0x4240
>
> Looks like no parts of it were written after initialization.
>
> I've also added just in case:
>
>  static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
>                              u32 error, bool valid)
>  {
> -       WARN_ON(vec > 0x1f);
> +       WARN_ON(vec > 0x1f || vec < 0);
>
> And it did _not_ fire.


It's this guy that passes NULL as exception to emulator_write_emulated:

static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt)
{
        struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
        char instruction[3];
        unsigned long rip = kvm_rip_read(vcpu);

        kvm_x86_ops->patch_hypercall(vcpu, instruction);

        return emulator_write_emulated(ctxt, rip, instruction, 3, NULL);
}

Mailed fix.



There is one more place that probably needs exception handling:

handle_vmread

     /* _system ok, as nested_vmx_check_permission verified cpl=0 */
    kvm_write_guest_virt_system(&vcpu->arch.emulate_ctxt, gva,
      &field_value, (is_long_mode(vcpu) ? 8 : 4), NULL);
  }

  nested_vmx_succeed(vcpu);
  return kvm_skip_emulated_instruction(vcpu);
}

If the write fails, it pretends that it succeed. SDM says:
  #PF(fault-code) If a page fault occurs in accessing a memory
destination operand.