Re: [RFC PATCH v4 07/28] sched: Add helper function to decide whether to allow cache aware scheduling

[Chen, Yu C]

On 10/2/2025 7:50 PM, Peter Zijlstra wrote:
> On Thu, Oct 02, 2025 at 07:31:40PM +0800, Chen, Yu C wrote:
> > On 10/1/2025 9:17 PM, Peter Zijlstra wrote:
> > > On Sat, Aug 09, 2025 at 01:03:10PM +0800, Chen Yu wrote:
> > > > From: Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>
> > > >
> > > > Cache-aware scheduling is designed to aggregate threads into their
> > > > preferred LLC, either via the task wake up path or the load balancing
> > > > path. One side effect is that when the preferred LLC is saturated,
> > > > more threads will continue to be stacked on it, degrading the workload's
> > > > latency. A strategy is needed to prevent this aggregation from going too
> > > > far such that the preferred LLC is too overloaded.
> > >
> > > So one of the ideas was to extend the preferred llc number to a mask.
> > > Update the preferred mask with (nr_threads / llc_size) bits, indicating
> > > the that many top llc as sorted by occupancy.
> >
> > Having more than one preferred LLC helps prevent aggregation from going
> > too far on a single preferred LLC.
> >
> > One question would be: if one LLC cannot hold all the threads of a process,
> > does a second preferred LLC help in this use case? Currently, this patch
> > gives up task aggregation and falls back to legacy load balancing if the
> > preferred LLC is overloaded. If we place threads across two preferred LLCs,
> > these threads might encounter cross-LLC latency anyway - so we may as well
> > let
> > legacy load balancing spread them out IMO.
>
> Well, being stuck on 2 LLCs instead of being spread across 10 still
> seems like a win, no?
>
> Remember, our friends at AMD have *MANY* LLCs.

I see, this makes sense.

> > Another issue that Patch 7 tries to address is avoiding task
> > bouncing between preferred LLCs and non-preferred LLCs. If we
> > introduce a preferred LLC priority list, logic to prevent task
> > bouncing between different preferred LLCs might be needed in
> > load balancing, which could become complicated.
>
> It doesn't really become more difficult to tell preferred LLC from
> non-preferred LLC with a asm. So why should things get more complicatd?

Besides distinguishing between preferred LLCs and non-preferred LLCs,
we might also want to distinguish between the ith preferred LLC and
the jth preferred LLC—this would help avoid task migration
bouncing between them by using hysteresis.

> Anyway, it was just one of the 'random' ideas I had kicking about.
> Reality always ruins things, *shrug* :-)

Yes, multiple preferred LLCs is a promising direction if the data
shows good results. We are planning to clean up the RFC patch and
send a refreshed version to summarize the current status. Meanwhile,
we will evaluate the multi-preferred LLC approach internally.
Thanks for providing this idea.

thanks,
Chenyu