Re: [PATCH v3] KVM: x86: Fix potential preemption when get the current kvmclock timestamp

[Wanpeng Li]

Ping,
2017-05-12 15:44 GMT+08:00 Paolo Bonzini <pbonzini@xxxxxxxxxx>:
>
>
> On 12/05/2017 03:12, Wanpeng Li wrote:
>> From: Wanpeng Li <wanpeng.li@xxxxxxxxxxx>
>>
>>  BUG: using __this_cpu_read() in preemptible [00000000] code: qemu-system-x86/2809
>>  caller is __this_cpu_preempt_check+0x13/0x20
>>  CPU: 2 PID: 2809 Comm: qemu-system-x86 Not tainted 4.11.0+ #13
>>  Call Trace:
>>   dump_stack+0x99/0xce
>>   check_preemption_disabled+0xf5/0x100
>>   __this_cpu_preempt_check+0x13/0x20
>>   get_kvmclock_ns+0x6f/0x110 [kvm]
>>   get_time_ref_counter+0x5d/0x80 [kvm]
>>   kvm_hv_process_stimers+0x2a1/0x8a0 [kvm]
>>   ? kvm_hv_process_stimers+0x2a1/0x8a0 [kvm]
>>   ? kvm_arch_vcpu_ioctl_run+0xac9/0x1ce0 [kvm]
>>   kvm_arch_vcpu_ioctl_run+0x5bf/0x1ce0 [kvm]
>>   kvm_vcpu_ioctl+0x384/0x7b0 [kvm]
>>   ? kvm_vcpu_ioctl+0x384/0x7b0 [kvm]
>>   ? __fget+0xf3/0x210
>>   do_vfs_ioctl+0xa4/0x700
>>   ? __fget+0x114/0x210
>>   SyS_ioctl+0x79/0x90
>>   entry_SYSCALL_64_fastpath+0x23/0xc2
>>  RIP: 0033:0x7f9d164ed357
>>   ? __this_cpu_preempt_check+0x13/0x20
>>
>> This can be reproduced by run kvm-unit-tests/hyperv_stimer.flat w/
>> CONFIG_PREEMPT and CONFIG_DEBUG_PREEMPT enabled.
>>
>> Safe access to per-CPU data requires a couple of constraints, though: the
>> thread working with the data cannot be preempted and it cannot be migrated
>> while it manipulates per-CPU variables. If the thread is preempted, the
>> thread that replaces it could try to work with the same variables; migration
>> to another CPU could also cause confusion. However there is no preemption
>> disable when reads host per-CPU tsc rate to calculate the current kvmclock
>> timestamp.
>>
>> This patch fixes it by utilizing get_cpu/put_cpu pair to guarantee both
>> __this_cpu_read() and rdtsc() are not preempted.
>>
>> Cc: Paolo Bonzini <pbonzini@xxxxxxxxxx>
>> Cc: Radim KrÄmÃÅ <rkrcmar@xxxxxxxxxx>
>> Signed-off-by: Wanpeng Li <wanpeng.li@xxxxxxxxxxx>
>> ---
>> v2 -> v3:
>>  * update patch description
>> v1 -> v2:
>>  * utilize get_cpu/put_cpu pair to protect both __this_cpu_read and rdtsc()
>>
>>  arch/x86/kvm/x86.c | 10 +++++++++-
>>  1 file changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>> index b54125b..3b5fc7e 100644
>> --- a/arch/x86/kvm/x86.c
>> +++ b/arch/x86/kvm/x86.c
>> @@ -1763,6 +1763,7 @@ u64 get_kvmclock_ns(struct kvm *kvm)
>>  {
>>       struct kvm_arch *ka = &kvm->arch;
>>       struct pvclock_vcpu_time_info hv_clock;
>> +     u64 ret;
>>
>>       spin_lock(&ka->pvclock_gtod_sync_lock);
>>       if (!ka->use_master_clock) {
>> @@ -1774,10 +1775,17 @@ u64 get_kvmclock_ns(struct kvm *kvm)
>>       hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset;
>>       spin_unlock(&ka->pvclock_gtod_sync_lock);
>>
>> +     /* both __this_cpu_read() and rdtsc() should be on the same cpu */
>> +     get_cpu();
>> +
>>       kvm_get_time_scale(NSEC_PER_SEC, __this_cpu_read(cpu_tsc_khz) * 1000LL,
>>                          &hv_clock.tsc_shift,
>>                          &hv_clock.tsc_to_system_mul);
>> -     return __pvclock_read_cycles(&hv_clock, rdtsc());
>> +     ret = __pvclock_read_cycles(&hv_clock, rdtsc());
>> +
>> +     put_cpu();
>> +
>> +     return ret;
>>  }
>>
>>  static void kvm_setup_pvclock_page(struct kvm_vcpu *v)
>>
>
> Reviewed-by: Paolo Bonzini <pbonzini@xxxxxxxxxx>
> Cc: stable@xxxxxxxxxxxxxxx