Re: [Question] Userspace throttling + "sched/fair: Combine detach into dequeue when migrating task" causes guest boot hang

[chenjinghuang]

On 8/26/2026 6:00 PM, Aaron Lu wrote:
> On Tue, Aug 25, 2026 at 12:06:29PM +0000, Chen Jinghuang wrote:
>> Hi, I'm seeing a VM boot hang on mainline, and I'd like to understand the
>> interaction between userspace throttling and the scheduler patch
>> "e1f078f50478 sched/fair: Combine detach into dequeue when migrating
>> task".
>>
>> Host:
>> aarch64, 96 CPUs (0-95), 4 NUMA nodes:
>> 	node0: 0-23, node1: 24-47, node2: 48-71, node3: 72-95
>> Mainline kernel tag: 7.2-rc1.
>>
>> Guest(libvirt/KVM) - described in words:
>> An aarch64 (virt-6.2) UEFI VM launched with `virsh create`; key config:
>>
>> - 128 vCPUs (statically placed, oversubscribed — the host has only 96
>>   physical CPUs).
>> - host-passthrough CPU model; GICv3; 64 GiB RAM.
>> - <cputune> has <global_quota> set to 400000; all <vcpupin> and
>>   <emulatorpin> entries are commented out, so there is no vCPU pinning.
>> - Storage: qcow2 on virtio-scsi (cache=none, io=native). HPET disabled.
>>
>> Userspace throttling:
>> The VM runs under a CPU-quota cap applied on the host. The actual values
>> from the cgroup controller are:
>>
>> 	cpu.cfs_period_us = 100000
>> 	cpu.cfs_quota_us  = 400000
>>
>> I also found that if I set cpu.cfs_quota_us to -1, or enlarge it beyond a
>> certain point, the guest boots fine.
>>
>> Symptom:
>> The guest hangs at some command early in boot and never reaches the login
>> prompt.
>>
> 
> I tried this on an x86 machine with v7.2-rc1 kernel and with quota set
> to 4 cpus, the VM booted fine; when I further reduced quota to 1 cpu, the
> guest kernel would dump a ton of soft lockups during boot. I also tried
> running an old 5.10 kernel(which doesn't have per-task throttle) and it
> behaved the same as v7.2-rc1.
> 
> The x86 machine has 64cores/128cpus and the VM I created has 128cpus and
> 128G memory.
> 
My host is an ARM64 machine without SMT, so 96 physical cores correspond to
96 logical CPUs. The VM is configured with 128 vCPUs, which is indeed a typical
CPU oversubscription scenario. In my machine, if I configure the VM with 96 vCPUs,
the issue don't occur either.

>> Observations:
>> Only reverting both of the following together makes it boot (neither one
>> alone suffices):
>>
>> 1. The kernel patch for userspace throttling.
>> 2. The scheduler patch:
>>    e1f078f50478 ("sched/fair: Combine detach into dequeue when migrating
>>    task")
> 
> I'm curious how you found e1f078f50478, just because it touched pelt?
> 
I located these two commits via git bisect:
- Comparison: On an older 5.10 kernel, the same test case (Quota set to 400000 with
  a 128-vCPU VM) boots completely fine, whereas on the mainline kernel, the guest hangs.
- Bisect steps: Without userspace throttling, git bisect pointed to commit e1f078f50478
  ("sched/fair: Combine detach into dequeue when migrating task"). Howevert, reverting
  e1f078f50478 alone on mainline v7.2 still resulted in a hang. Futher bisecting led
  to the userspace throttling patch("sched/fair: Switch to task based throttle model").
  I found that only reverting both e1f078f50478 and the userspace throttling ptach together
  restores normal guest boot.

>>
>> Reverting only one of them still hangs; reverting both together boots fine.
>>
> 
> On top of v7.2-rc1, right?
> 
Yes, the previous test was based on v7.2-rc1. You can also try reproducing it on top
of the official v7.2 tag (commit 8d3ae59288f1).

>> Question:
>> I don't fully understand how these two interact. My rough guess: e1f078f50478
>> ("sched/fair: Combine detach into dequeue when migrating task") affects the
>> PELT accounting, and the userspace throttling also has logic that affects PELT
>> accounting. When both are combined, load balancing and subsequent scheduling
>> behavior may end up misbehaving, stalling the guest.
> 
> Is the host busy? If the host has many idle cpus, even the pelt is
> wrecked(which I doubt), it should not cause the qemu task being starved.
> The PELT accounting matters when tasks have to compet the same CPU, but
> if your host system has many idle cpus, that should not happen.
> 
The host is not running any other workloads besides the tasks associated with starting the
VM. However, because this is an oversubscribed setup, tasks are running across all 96 host
CPUs(even though the utilization on most CPUs is below 10%), so the host don't have many
idle CPUs available.

> And from the log you posted for the cpu usage, it appears that task
> group is getting cpu time.

It seems possible that the task gets throttled shortly after receiving a small time slice
in each period.