[PATCH RFC 0/2] Fix KVM guest scheduling accounting issue related to stealtime

[Dongli Zhang]

This RFC fixes a KVM guest scheduling accounting issue.

The Linux scheduler may allow one CPU to perform accounting for another CPU
through update_rq_clock_task(). This works well on baremetal.

However, in a KVM guest, stealtime may not be accounted for and deducted
from runtime correctly.

Suppose vCPU A performs accounting for vCPU B, while vCPU B is preempted
and stalled by the KVM host for 10 seconds. Unfortunately, the KVM host
does not update stealtime until vCPU B is about to re-enter the guest. As a
result, vCPU A cannot observe the increase in vCPU B's stealtime.

Consequently, the guest kernel incorrectly considers a task running on
vCPU B during the stall to have exclusively used the vCPU for an extended
period. The task may therefore incur an additional scheduling penalty.

Later, when stealtime is updated, it may exceed the elapsed runtime delta,
preventing all of the stealtime from being deducted.

To fix this issue, update KVM stealtime before clearing the vCPU's
preempted state. In the guest scheduler, use vcpu_is_preempted() to help
defer clock_task updates performed by a remote CPU while the owning vCPU is
reported as preempted.


To reproduce, first configure KVM host pCPU 10 so that a vCPU thread pinned to
it can be stalled.

hv# echo -1 | sudo tee /proc/sys/kernel/sched_rt_runtime_us

hv# echo 0 | sudo tee /sys/kernel/debug/sched/fair_server/cpu10/runtime

hv# cat /sys/kernel/debug/sched/fair_server/cpu10/runtime
0

hv# cat /sys/kernel/debug/sched/fair_server/cpu10/period
1000000000


Here is the QEMU command line. The guest kernel has below configs.

CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_CLOCK=y

qemu-system-x86_64 \
-machine q35,kernel_irqchip=split \
-accel kvm -cpu host \
-smp 4 -m 8G \
-hda boot.qcow2 \
-monitor stdio -vnc :8 \
-net nic -net user,hostfwd=tcp::5028-:22 \
-kernel mainline-linux/arch/x86_64/boot/bzImage \
-append "root=/dev/sda1 init=/sbin/init text loglevel=7 console=ttyS0"


Pin vCPU 2 to pCPU 10.

(qemu) info cpus
* CPU #0: thread_id=12525 model=host
  CPU #1: thread_id=12526 model=host
  CPU #2: thread_id=12528 model=host
  CPU #3: thread_id=12529 model=host

hv$ sudo taskset -pc 10 12528


Run runtime_stall_detector_v2.py in the guest VM. It creates five threads
pinned to vCPU 2. Each thread periodically reads the clocksource and
detects sudden forward jumps in time, indicating that the thread was
previously stalled. The source code is appended to the end of this cover
letter email.

[root@vm ~]# ./runtime_stall_detector_v2.py 2 5
started 5 tasks on guest cpu 2
worker pids: 441 442 443 444 445
trigger host-side vCPU starvation now
worker=3 pid=443 pinned_cpu=2
worker=1 pid=441 pinned_cpu=2
worker=2 pid=442 pinned_cpu=2
worker=4 pid=444 pinned_cpu=2
worker=5 pid=445 pinned_cpu=2

Now run the following command on the KVM host to preempt and stall threads
on pCPU 10. As a result, vCPU 2 and the five threads pinned to it will also
be stalled for 10 seconds.

hv$ sudo timeout 10s taskset -c 10 chrt -f 90 bash -c 'while :; do :; done'


Ideally, we expect each of the five threads to observe a 10-second forward
jump. However, one thread running on vCPU 2 during the stall will incur an
additional penalty.

[root@vm ~]# ./runtime_stall_detector_v2.py 2 5
... ...
1787476745.895051479: worker=5 pid=445 gap=9.997750 sec
1787476745.895614147: worker=1 pid=441 gap=10.004634 sec
1787476745.898544550: worker=2 pid=442 gap=10.005462 sec
1787476745.901545286: worker=4 pid=444 gap=10.006361 sec
1787476785.617044926: worker=3 pid=443 gap=49.718478 sec --> additional penalty!


The issue is no longer reproducible when the patchset is applied to both
the KVM guest and host.

1787507053.410693884: worker=4 pid=436 gap=9.995703 sec
1787507053.411839247: worker=5 pid=437 gap=10.004244 sec
1787507053.413943768: worker=2 pid=434 gap=10.002146 sec
1787507053.414848804: worker=1 pid=433 gap=10.005150 sec
1787507053.419051886: worker=3 pid=435 gap=10.005138 sec


Dongli Zhang (2):
  KVM: x86: Update stealtime before clearing preempted state
  sched/core: Defer preempted remote vCPU task clock updates

 arch/x86/kvm/x86.c   | 58 ++++++++++++++++++++++++++---------------------
 kernel/sched/core.c  | 24 ++++++++++++++++++++
 kernel/sched/sched.h |  1 +
 3 files changed, 57 insertions(+), 26 deletions(-)

base-commit: 818bebeb63dd6bf5f4e07e145f6cdbace520a34c

Thank you very much!

Dongli Zhang


----------------------

[root@vm ~]# cat runtime_stall_detector_v2.py
#!/usr/bin/env python3
import multiprocessing as mp
import os
import signal
import subprocess
import sys
import time


def worker(cpu, index, done):
    os.sched_setaffinity(0, {cpu})
    pid = os.getpid()

    print(f"worker={index} pid={pid} pinned_cpu={cpu}", flush=True)

    prev = time.monotonic_ns()
    while True:
        now = time.monotonic_ns()
        gap = (now - prev) / 1e9

        if gap > 0.5:
            print(f"{time.time():.9f}: worker={index} pid={pid} gap={gap:.6f} sec", flush=True)
            if gap > 30:
                done.set()
                return

        prev = now


def kill_old_detectors():
    subprocess.run(["pkill", "-f", "runtime_stall_detector.py"], check=False)


def main():
    cpu = int(sys.argv[1]) if len(sys.argv) > 1 else 2
    nr = int(sys.argv[2]) if len(sys.argv) > 2 else 5

    kill_old_detectors()

    done = mp.Event()
    procs = [mp.Process(target=worker, args=(cpu, i, done)) for i in range(1, nr + 1)]

    for proc in procs:
        proc.start()

    print(f"started {nr} tasks on guest cpu {cpu}", flush=True)
    print("worker pids:", " ".join(str(proc.pid) for proc in procs), flush=True)
    print("trigger host-side vCPU starvation now", flush=True)

    try:
        while not done.wait(1):
            pass
    except KeyboardInterrupt:
        pass
    finally:
        for proc in procs:
            if proc.is_alive():
                os.kill(proc.pid, signal.SIGTERM)
        for proc in procs:
            proc.join()


if __name__ == "__main__":
    main()