Re: [PATCH v3 15/16] kvm: x86: ioapic: Lazy update IOAPIC EOI

[Suthikulpanit, Suravee]

Paolo,

On 10/9/19 4:21 AM, Paolo Bonzini wrote:
> On 13/09/19 21:01, Suthikulpanit, Suravee wrote:
>>   	/*
>> +	 * In case APICv accelerate EOI write and do not trap,
>> +	 * in-kernel IOAPIC will not be able to receive the EOI.
>> +	 * In this case, we do lazy update of the pending EOI when
>> +	 * trying to set IOAPIC irq.
>> +	 */
>> +	if (kvm_apicv_eoi_accelerate(ioapic->kvm, edge))
>> +		ioapic_lazy_update_eoi(ioapic, irq);
>> +
> 
> This is okay for the RTC, and in fact I suggest that you make it work
> like this even for Intel.  This will get rid of kvm_apicv_eoi_accelerate
> and be nicer overall.
> 
> However, it cannot work for the in-kernel PIT, because it is currently
> checking ps->irq_ack before kvm_set_irq.  Unfortunately, the in-kernel
> PIT is relying on the ack notifier to timely queue the pt->worker work
> item and reinject the missed tick.
> 
> Thus, you cannot enable APICv if ps->reinject is true.
> 
> Perhaps you can make kvm->arch.apicv_state a disabled counter?  Then
> Hyper-V can increment it when enabled, PIT can increment it when
> ps->reinject becomes true and decrement it when it becomes false;
> finally, svm.c can increment it when an SVM guest is launched and
> increment/decrement it around ExtINT handling?

I have been looking into the disabled counter idea and found a couple 
issues:

* I am seeing more calls to enable_irq_window() than the number of 
interrupt_window_interception(). This results in imbalanced 
increment/decrement of the counter.

* APICv can be deactivated due to several reasons. Currently, it is 
difficult to figure out why, and this makes debugging APICv difficult.

What if we change kvm->arch.apicv_state to kvm->arch.apicv_disable_mask 
and have each bit representing the reason for deactivating APICv.

For example:
     #define APICV_DISABLE_MASK_IRQWIN        0
     #define APICV_DISABLE_MASK_HYPERV        1
     #define APICV_DISABLE_MASK_PIT_REINJ     2
     #define APICV_DISABLE_MASK_NESTED        3

In this case, we activate APICv only if kvm->arch.apicv_disable_mask == 
0. This way, we can find out why APICv is deactivated on a particular VM 
at a particular point in time.

Thanks,
Suravee