Re: [PATCH RFC V9 0/19] Paravirtualized ticket spinlocks
From: Raghavendra K T
Date: Wed Jun 26 2013 - 09:36:48 EST
On 06/26/2013 06:22 PM, Gleb Natapov wrote:
On Wed, Jun 26, 2013 at 01:37:45PM +0200, Andrew Jones wrote:
On Wed, Jun 26, 2013 at 02:15:26PM +0530, Raghavendra K T wrote:
On 06/25/2013 08:20 PM, Andrew Theurer wrote:
On Sun, 2013-06-02 at 00:51 +0530, Raghavendra K T wrote:
This series replaces the existing paravirtualized spinlock mechanism
with a paravirtualized ticketlock mechanism. The series provides
implementation for both Xen and KVM.
Changes in V9:
- Changed spin_threshold to 32k to avoid excess halt exits that are
causing undercommit degradation (after PLE handler improvement).
- Added kvm_irq_delivery_to_apic (suggested by Gleb)
- Optimized halt exit path to use PLE handler
V8 of PVspinlock was posted last year. After Avi's suggestions to look
at PLE handler's improvements, various optimizations in PLE handling
have been tried.
Sorry for not posting this sooner. I have tested the v9 pv-ticketlock
patches in 1x and 2x over-commit with 10-vcpu and 20-vcpu VMs. I have
tested these patches with and without PLE, as PLE is still not scalable
with large VMs.
Hi Andrew,
Thanks for testing.
System: x3850X5, 40 cores, 80 threads
1x over-commit with 10-vCPU VMs (8 VMs) all running dbench:
----------------------------------------------------------
Total
Configuration Throughput(MB/s) Notes
3.10-default-ple_on 22945 5% CPU in host kernel, 2% spin_lock in guests
3.10-default-ple_off 23184 5% CPU in host kernel, 2% spin_lock in guests
3.10-pvticket-ple_on 22895 5% CPU in host kernel, 2% spin_lock in guests
3.10-pvticket-ple_off 23051 5% CPU in host kernel, 2% spin_lock in guests
[all 1x results look good here]
Yes. The 1x results look too close
2x over-commit with 10-vCPU VMs (16 VMs) all running dbench:
-----------------------------------------------------------
Total
Configuration Throughput Notes
3.10-default-ple_on 6287 55% CPU host kernel, 17% spin_lock in guests
3.10-default-ple_off 1849 2% CPU in host kernel, 95% spin_lock in guests
3.10-pvticket-ple_on 6691 50% CPU in host kernel, 15% spin_lock in guests
3.10-pvticket-ple_off 16464 8% CPU in host kernel, 33% spin_lock in guests
I see 6.426% improvement with ple_on
and 161.87% improvement with ple_off. I think this is a very good sign
for the patches
[PLE hinders pv-ticket improvements, but even with PLE off,
we still off from ideal throughput (somewhere >20000)]
Okay, The ideal throughput you are referring is getting around atleast
80% of 1x throughput for over-commit. Yes we are still far away from
there.
1x over-commit with 20-vCPU VMs (4 VMs) all running dbench:
----------------------------------------------------------
Total
Configuration Throughput Notes
3.10-default-ple_on 22736 6% CPU in host kernel, 3% spin_lock in guests
3.10-default-ple_off 23377 5% CPU in host kernel, 3% spin_lock in guests
3.10-pvticket-ple_on 22471 6% CPU in host kernel, 3% spin_lock in guests
3.10-pvticket-ple_off 23445 5% CPU in host kernel, 3% spin_lock in guests
[1x looking fine here]
I see ple_off is little better here.
2x over-commit with 20-vCPU VMs (8 VMs) all running dbench:
----------------------------------------------------------
Total
Configuration Throughput Notes
3.10-default-ple_on 1965 70% CPU in host kernel, 34% spin_lock in guests
3.10-default-ple_off 226 2% CPU in host kernel, 94% spin_lock in guests
3.10-pvticket-ple_on 1942 70% CPU in host kernel, 35% spin_lock in guests
3.10-pvticket-ple_off 8003 11% CPU in host kernel, 70% spin_lock in guests
[quite bad all around, but pv-tickets with PLE off the best so far.
Still quite a bit off from ideal throughput]
This is again a remarkable improvement (307%).
This motivates me to add a patch to disable ple when pvspinlock is on.
probably we can add a hypercall that disables ple in kvm init patch.
but only problem I see is what if the guests are mixed.
(i.e one guest has pvspinlock support but other does not. Host
supports pv)
How about reintroducing the idea to create per-kvm ple_gap,ple_window
state. We were headed down that road when considering a dynamic window at
one point. Then you can just set a single guest's ple_gap to zero, which
would lead to PLE being disabled for that guest. We could also revisit
the dynamic window then.
Can be done, but lets understand why ple on is such a big problem. Is it
possible that ple gap and SPIN_THRESHOLD are not tuned properly?
The one obvious reason I see is commit awareness inside the guest. for
under-commit there is no necessity to do PLE, but unfortunately we do.
atleast we return back immediately in case of potential undercommits,
but we still incur vmexit delay.
same applies to SPIN_THRESHOLD. SPIN_THRESHOLD should be ideally more
for undercommit and less for overcommit.
with this patch series SPIN_THRESHOLD is increased to 32k to solely
avoid under-commit regressions but it would have eaten some amount of
overcommit performance.
In summary: excess halt-exit/pl-exit was one main reason for
undercommit regression. (compared to pl disabled case)
1. dynamic ple window was one solution for PLE, which we can experiment
further. (at VM level or global).
The other experiment I was thinking is to extend spinlock to
accommodate vcpuid (Linus has opposed that but may be worth a
try).
2. Andrew Theurer had patch to reduce double runq lock that I will be
testing.
I have some older experiments to retry though they did not give
significant improvements before the PLE handler modified.
Andrew, do you have any other details to add (from perf report that you
usually take with these experiments)?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/