is there a known change in CPU affinity for new containerized tasks between 5.10 and 6.6?

[Chris Friesen]

Hi all,

I'm not subscribed to the list so please CC me on replies.

I'm seeing some changes in CPU affinity for tasks in new Kubernetes 
containers when going from the 5.10 kernel to the 6.6 kernel, and was 
wondering if the new behaviour is expected or would be considered a 
regression.

I'm running Kubernetes 1.28.4, with containerd and runc.   Kubelet is 
running with "--reserved-cpus=0,64 --cpu-manager-policy=none", and the 
containerd daemon is affined to CPUs 0 and 64.

When using the 5.10 kernel when I start up a new pod the tasks within 
the new pod/container are affined to all online CPUs.

When using the 6.6.7 kernel, when I start up a new pod the tasks within 
the pod/container are affined to just CPUs 0 and 64, even though the 
cgroup for the container specifies "cpuset.cpus" and 
"cpuset.effective_cpus" values corresponding to all the online CPUs.

In the example below, pid 220350 is running within the new container.


[sysadmin@controller-0 ~(keystone_admin)]$ taskset -c -p 220350
pid 220350's current affinity list: 0,64

root@controller-0:/var/home/sysadmin# cat /proc/220350/cpuset
/k8s-infra/kubepods/besteffort/pod1c4b3a1b-5c63-4f38-b568-57ce87c9c3a0/c2b5de13e14d13ae3f6f8fb39bd287579261f00a1a2a0fdca3e764fbfbef28a7

root@controller-0:/var/home/sysadmin# cat 
/sys/fs/cgroup/cpuset/k8s-infra/kubepods/besteffort/pod1c4b3a1b-5c63-4f38-b568-57ce87c9c3a0/c2b5de13e14d13ae3f6f8fb39bd287579261f00a1a2a0fdca3e764fbfbef28a7/cpuset.cpus
0-127


I can explicitly re-affine pid 220350 to CPUs 0-127 and the kernel 
allows it.

If I affine the containerd daemon to all online CPUs and start up a new 
pod, the tasks within the new container are affined to all online CPUs.

It seems like with the newer kernel something has changed to cause the 
tasks within the container to inherit the CPU affinity of the parent 
containerd daemon rather than being affined to the CPU affinity 
specified for the container.  Anyone have any ideas what might be 
causing this?

Thanks,
Chris