Re: hotplug support for arch/arc/plat-eznps platform

[Peter Zijlstra]

On Wed, Aug 09, 2017 at 03:19:02PM +0000, Ofer Levi(SW) wrote:
> I appreciate your effort and detailed reply, however I'm still experiencing  performance hit at 
> partition_sched_domains(). It seems the issue is due to the large magnitude of cpus. 
> I used he suggested method 2, patched in the diffs and used the command line switch isolcpus to
> kill load-balancing.
> It did save few hundredth of a sec per cpu. When I limited number of available cpus 
> (using present and possible cpus ) to 48, it did reduced dramatically this function execution time:
> 
> With 4K available cpus :
> [   48.890000] ## CPU16 LIVE ##: Executing Code...
> [   48.910000] partition_sched_domains start
> [   49.360000] partition_sched_domains end
> 
> With 48 available cpus:
> [   36.950000] ## CPU16 LIVE ##: Executing Code...
> [   36.950000] partition_sched_domains start
> [   36.960000] partition_sched_domains end
> 
> Note that I currently use kernel version: 4.8.0.17.0600.00.0000, if this has any influence.
> Would appreciate your thoughts.
> 

Does something like this cure things? It seems we're doing a
possible_cpus iteration for sysctl cruft, and that will most certainly
hurt on your little toy :-)

Not sure what the more generic solution to that would be, but the below
avoids it for isolcpus.

---
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -85,6 +85,7 @@ int sysctl_sched_rt_runtime = 950000;
 
 /* CPUs with isolated domains */
 cpumask_var_t cpu_isolated_map;
+cpumask_var_t non_isolated_cpus;
 
 /*
  * __task_rq_lock - lock the rq @p resides on.
@@ -5685,8 +5686,6 @@ static inline void sched_init_smt(void)
 
 void __init sched_init_smp(void)
 {
-	cpumask_var_t non_isolated_cpus;
-
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 
 	sched_init_numa();
@@ -5697,17 +5696,17 @@ void __init sched_init_smp(void)
 	 * happen.
 	 */
 	mutex_lock(&sched_domains_mutex);
-	sched_init_domains(cpu_active_mask);
 	cpumask_andnot(non_isolated_cpus, cpu_possible_mask, cpu_isolated_map);
 	if (cpumask_empty(non_isolated_cpus))
 		cpumask_set_cpu(smp_processor_id(), non_isolated_cpus);
+
+	sched_init_domains(cpu_active_mask);
 	mutex_unlock(&sched_domains_mutex);
 
 	/* Move init over to a non-isolated CPU */
 	if (set_cpus_allowed_ptr(current, non_isolated_cpus) < 0)
 		BUG();
 	sched_init_granularity();
-	free_cpumask_var(non_isolated_cpus);
 
 	init_sched_rt_class();
 	init_sched_dl_class();
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -327,6 +327,8 @@ static struct ctl_table *sd_alloc_ctl_cp
 	return table;
 }
 
+extern cpumask_var_t non_isolated_cpus;
+
 static struct ctl_table_header *sd_sysctl_header;
 void register_sched_domain_sysctl(void)
 {
@@ -340,7 +342,7 @@ void register_sched_domain_sysctl(void)
 	if (entry == NULL)
 		return;
 
-	for_each_possible_cpu(i) {
+	for_each_cpu(i, non_isolated_cpus) {
 		snprintf(buf, 32, "cpu%d", i);
 		entry->procname = kstrdup(buf, GFP_KERNEL);
 		entry->mode = 0555;