Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs

[Tim Chen]

On Mon, 2025-10-27 at 20:56 +0800, Chen, Yu C wrote:
> Hi Prateek,
> 
> On 10/27/2025 1:42 PM, K Prateek Nayak wrote:
> > Hello Tim,
> > 
> > On 10/11/2025 11:54 PM, Tim Chen wrote:
> > > @@ -2530,10 +2531,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
> > >   				 * between LLCs and memory channels.
> > >   				 */
> > >   				nr_llcs = sd->span_weight / child->span_weight;
> > > -				if (nr_llcs == 1)
> > > +				if (nr_llcs == 1) {
> > >   					imb = sd->span_weight >> 3;
> > > -				else
> > > +				} else {
> > >   					imb = nr_llcs;
> > > +					has_multi_llcs = true;
> > 
> > One caution: this will not hold if all the CPUs aren't online during boot.
> > One case I can think of is when the kernel is booted with "maxcpus" cmdline
> > and CPUs are hotplugged later.
> > 
> > Unfortunately, I don't think we even have the raw topology data from the
> > arch/ side under such scenario to accurately make a call if the system
> > contains single or multiple LLC :(
> > 
> > I'm not sure if it is feasible but assuming the task_work() cannot run if
> > &sched_cache_allowed is false, can the fist instance of the task work for
> > sched_cache do the necessary setup?
> > 
> 
> build_sched_domains() might get invoked to rebuild the corresponding sched
> domains during CPU hotplug via cpuset subsystem. So if the CPU gets online
> after bootup, we still have the chance to detect multiple LLCs I suppose?

The case Pratek brought up of adding CPUs and enabling SCHED_CACHE 
should be covered. 

The trickier case is if we disable SCHED_CACHE when CPUs are
offlined and multi_cpus becomes false.  We'll need to clear out rq->nr_pref_llcs
data and tasks' preferred LLC would need to be cleared.  Or else the accounting
could be skewed we bring CPU online later and again re-enable SCHED_CACHE.
So far we haven't done that when we disable SCHED_CACHE from an enabled state.

Tim

> 
> I did a check on my VM:
> root@ubuntu:/sys/devices/system/cpu# lscpu
> CPU(s):                      32
>    On-line CPU(s) list:       0-7
> root@ubuntu:/sys/devices/system/cpu# echo 1 > cpu31/online
> Tracing ... Hit Ctrl-C to end.
> ^C
> 
> @build_sched_domains[
>      build_sched_domains+5
>      partition_sched_domains+613
>      cpuset_update_active_cpus+838
>      sched_cpu_activate+272
>      cpuhp_invoke_callback+340
>      cpuhp_thread_fun+139
>      smpboot_thread_fn+238
>      kthread+249
>      ret_from_fork+193
>      ret_from_fork_asm+26
> ]: 1
> 
> thanks,
> Chenyu