Re: [PATCH 2/2] x86/idle: use dynamic halt poll

[Alexander Graf]

On 13.07.17 13:49, Yang Zhang wrote:
> On 2017/7/4 22:13, Radim KrÄmÃÅ wrote:
> > 2017-07-03 17:28+0800, Yang Zhang:
> > > The background is that we(Alibaba Cloud) do get more and more complaints
> > > from our customers in both KVM and Xen compare to bare-mental.After
> > > investigations, the root cause is known to us: big cost in message 
> > > passing
> > > workload(David show it in KVM forum 2015)
> > >
> > > A typical message workload like below:
> > > vcpu 0                             vcpu 1
> > > 1. send ipi                     2.  doing hlt
> > > 3. go into idle                 4.  receive ipi and wake up from hlt
> > > 5. write APIC time twice        6.  write APIC time twice to
> > >    to stop sched timer              reprogram sched timer
> >
> > One write is enough to disable/re-enable the APIC timer -- why does
> > Linux use two?
>
> One is to remove the timer and another one is to reprogram the timer. 
> Normally, only one write to remove the timer.But in some cases, it will 
> reprogram it.
>
> > > 7. doing hlt                    8.  handle task and send ipi to
> > >                                     vcpu 0
> > > 9. same to 4.                   10. same to 3
> > >
> > > One transaction will introduce about 12 vmexits(2 hlt and 10 msr 
> > > write). The
> > > cost of such vmexits will degrades performance severely.
> >
> > Yeah, sounds like too much ... I understood that there are
> >
> >   IPI from 1 to 2
> >   4 * APIC timer
> >   IPI from 2 to 1
> >
> > which adds to 6 MSR writes -- what are the other 4?
>
> In the worst case, each timer will touch APIC timer twice.So it will add 
> additional 4 msr writse. But this is  not always true.
>
> > >                                                          Linux kernel
> > > already provide idle=poll to mitigate the trend. But it only 
> > > eliminates the
> > > IPI and hlt vmexit. It has nothing to do with start/stop sched timer. A
> > > compromise would be to turn off NOHZ kernel, but it is not the default
> > > config for new distributions. Same for halt-poll in KVM, it only 
> > > solve the
> > > cost from schedule in/out in host and can not help such workload much.
> > >
> > > The purpose of this patch we want to improve current idle=poll 
> > > mechanism to
> >
> > Please aim to allow MWAIT instead of idle=poll -- MWAIT doesn't slow
> > down the sibling hyperthread.  MWAIT solves the IPI problem, but doesn't
> > get rid of the timer one.
>
> Yes, i can try it. But MWAIT will not yield CPU, it only helps the 
> sibling hyperthread as you mentioned.

If you implement proper MWAIT emulation that conditionally gets en- or 
disabled depending on the same halt poll dynamics that we already have 
for in-host HLT handling, it will also yield the CPU.

As for the timer - are you sure the problem is really the overhead of 
the timer configuration, not the latency that it takes to actually fire 
the guest timer?

One major problem I see is that we configure the host hrtimer to fire at 
the point in time when the guest wants to see a timer event. But in a 
virtual environment, the point in time when we have to start switching 
to the VM really should be a bit *before* the guest wants to be woken 
up, as it takes quite some time to switch back into the VM context.


Alex