Re: [patch 3/4] isolcpus: affine kernel threads to housekeeping cpus

From: Randy Dunlap
Date: Wed Apr 01 2020 - 18:32:53 EST


On 4/1/20 5:10 AM, Marcelo Tosatti wrote:
> This is a kernel enhancement that configures the cpu affinity of kernel
> threads via kernel boot option nohz_full=.
>
> When this option is specified, the cpumask is immediately applied upon
> thread launch. This does not affect kernel threads that specify cpu
> and node.
>
> This allows CPU isolation (that is not allowing certain threads
> to execute on certain CPUs) without using the isolcpus=domain parameter,
> making it possible to enable load balancing on such CPUs
> during runtime (see kernel-parameters.txt).
>
> Note-1: this is based off on Wind River's patch at
> https://github.com/starlingx-staging/stx-integ/blob/master/kernel/kernel-std/centos/patches/affine-compute-kernel-threads.patch
>
> Difference being that this patch is limited to modifying
> kernel thread cpumask: Behaviour of other threads can
> be controlled via cgroups or sched_setaffinity.
>
> Note-2: Wind River's patch was based off Christoph Lameter's patch at
> https://lwn.net/Articles/565932/ with the only difference being
> the kernel parameter changed from kthread to kthread_cpus.
>
> Signed-off-by: Marcelo Tosatti <mtosatti@xxxxxxxxxx>
>

Hi Marcelo,

> ---
> Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++

Patch is missing those Docum bits.

> include/linux/sched/isolation.h | 1 +
> kernel/kthread.c | 6 ++++--
> kernel/sched/isolation.c | 6 ++++++
> 4 files changed, 19 insertions(+), 2 deletions(-)
>
> Index: linux-2.6/include/linux/sched/isolation.h
> ===================================================================
> --- linux-2.6.orig/include/linux/sched/isolation.h
> +++ linux-2.6/include/linux/sched/isolation.h
> @@ -14,6 +14,7 @@ enum hk_flags {
> HK_FLAG_DOMAIN = (1 << 5),
> HK_FLAG_WQ = (1 << 6),
> HK_FLAG_MANAGED_IRQ = (1 << 7),
> + HK_FLAG_KTHREAD = (1 << 8),
> };
>
> #ifdef CONFIG_CPU_ISOLATION
> Index: linux-2.6/kernel/kthread.c
> ===================================================================
> --- linux-2.6.orig/kernel/kthread.c
> +++ linux-2.6/kernel/kthread.c
> @@ -23,6 +23,7 @@
> #include <linux/ptrace.h>
> #include <linux/uaccess.h>
> #include <linux/numa.h>
> +#include <linux/sched/isolation.h>
> #include <trace/events/sched.h>
>
> static DEFINE_SPINLOCK(kthread_create_lock);
> @@ -347,7 +348,8 @@ struct task_struct *__kthread_create_on_
> * The kernel thread should not inherit these properties.
> */
> sched_setscheduler_nocheck(task, SCHED_NORMAL, &param);
> - set_cpus_allowed_ptr(task, cpu_possible_mask);
> + set_cpus_allowed_ptr(task,
> + housekeeping_cpumask(HK_FLAG_KTHREAD));
> }
> kfree(create);
> return task;
> @@ -572,7 +574,7 @@ int kthreadd(void *unused)
> /* Setup a clean context for our children to inherit. */
> set_task_comm(tsk, "kthreadd");
> ignore_signals(tsk);
> - set_cpus_allowed_ptr(tsk, cpu_possible_mask);
> + set_cpus_allowed_ptr(tsk, housekeeping_cpumask(HK_FLAG_KTHREAD));
> set_mems_allowed(node_states[N_MEMORY]);
>
> current->flags |= PF_NOFREEZE;
> Index: linux-2.6/kernel/sched/isolation.c
> ===================================================================
> --- linux-2.6.orig/kernel/sched/isolation.c
> +++ linux-2.6/kernel/sched/isolation.c
> @@ -141,7 +141,7 @@ static int __init housekeeping_nohz_full
> unsigned int flags;
>
> flags = HK_FLAG_TICK | HK_FLAG_WQ | HK_FLAG_TIMER | HK_FLAG_RCU |
> - HK_FLAG_MISC | HK_FLAG_SCHED;
> + HK_FLAG_MISC | HK_FLAG_SCHED | HK_FLAG_KTHREAD;
>
> return housekeeping_setup(str, flags);
> }
>
>

thanks.
--
~Randy