is there a known change in CPU affinity for new containerized tasks between 5.10 and 6.6?

From: Chris Friesen
Date: Wed Jun 19 2024 - 13:11:13 EST


Hi all,

I'm not subscribed to the list so please CC me on replies.

I'm seeing some changes in CPU affinity for tasks in new Kubernetes containers when going from the 5.10 kernel to the 6.6 kernel, and was wondering if the new behaviour is expected or would be considered a regression.

I'm running Kubernetes 1.28.4, with containerd and runc. Kubelet is running with "--reserved-cpus=0,64 --cpu-manager-policy=none", and the containerd daemon is affined to CPUs 0 and 64.

When using the 5.10 kernel when I start up a new pod the tasks within the new pod/container are affined to all online CPUs.

When using the 6.6.7 kernel, when I start up a new pod the tasks within the pod/container are affined to just CPUs 0 and 64, even though the cgroup for the container specifies "cpuset.cpus" and "cpuset.effective_cpus" values corresponding to all the online CPUs.

In the example below, pid 220350 is running within the new container.


[sysadmin@controller-0 ~(keystone_admin)]$ taskset -c -p 220350
pid 220350's current affinity list: 0,64

root@controller-0:/var/home/sysadmin# cat /proc/220350/cpuset
/k8s-infra/kubepods/besteffort/pod1c4b3a1b-5c63-4f38-b568-57ce87c9c3a0/c2b5de13e14d13ae3f6f8fb39bd287579261f00a1a2a0fdca3e764fbfbef28a7

root@controller-0:/var/home/sysadmin# cat /sys/fs/cgroup/cpuset/k8s-infra/kubepods/besteffort/pod1c4b3a1b-5c63-4f38-b568-57ce87c9c3a0/c2b5de13e14d13ae3f6f8fb39bd287579261f00a1a2a0fdca3e764fbfbef28a7/cpuset.cpus
0-127


I can explicitly re-affine pid 220350 to CPUs 0-127 and the kernel allows it.

If I affine the containerd daemon to all online CPUs and start up a new pod, the tasks within the new container are affined to all online CPUs.

It seems like with the newer kernel something has changed to cause the tasks within the container to inherit the CPU affinity of the parent containerd daemon rather than being affined to the CPU affinity specified for the container. Anyone have any ideas what might be causing this?

Thanks,
Chris