RE: [RFC PATCH v2 02/23] sched/topology: Introduce a NUMA distance matrix with unique distance values

[Jianyong Wu]

Hi Chenyu,

> -----Original Message-----
> From: Chen Yu <yu.c.chen@xxxxxxxxx>
> Sent: Friday, October 9, 2026 12:35 AM
> To: Jianyong Wu <wujianyong@xxxxxxxx>
> Cc: Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>; Peter Zijlstra
> <peterz@xxxxxxxxxxxxx>; Ingo Molnar <mingo@xxxxxxxxxx>; Juri Lelli
> <juri.lelli@xxxxxxxxxx>; Vincent Guittot <vincent.guittot@xxxxxxxxxx>;
> Dietmar Eggemann <dietmar.eggemann@xxxxxxx>; Steven Rostedt
> <rostedt@xxxxxxxxxxx>; Ben Segall <bsegall@xxxxxxxxxx>; Mel Gorman
> <mgorman@xxxxxxx>; Valentin Schneider <vschneid@xxxxxxxxxx>; K
> Prateek Nayak <kprateek.nayak@xxxxxxx>; Shrikanth Hegde
> <sshegde@xxxxxxxxxxxxx>; Phil Auld <pauld@xxxxxxxxxx>; Andrew
> Morton <akpm@xxxxxxxxxxxxxxxxxxxx>; David Hildenbrand
> <david@xxxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx; linux-mm@xxxxxxxxx;
> jianyong.wu@xxxxxxxxxxx; Yuan Zhong <zhongyuan@xxxxxxxx>; Huangsj
> <huangsj@xxxxxxxx>; Fengyu Wang <wangfengyu@xxxxxxxx>; Zhiwei Ying
> <yingzhiwei@xxxxxxxx>; justin.he@xxxxxxx; chen.yu@xxxxxxxxx
> Subject: Re: [RFC PATCH v2 02/23] sched/topology: Introduce a NUMA
> distance matrix with unique distance values
> 
> Hi Jianyong,
> 
> On Thu, Oct 08, 2026 at 07:35:10AM +0000, Jianyong Wu wrote:
> > Hi Tim,
> >
> > > -----Original Message-----
> > > From: Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx>
> > > Sent: Friday, October 2, 2026 6:04 AM
> > > To: Jianyong Wu <wujianyong@xxxxxxxx>; Peter Zijlstra
> > > <peterz@xxxxxxxxxxxxx>
> > > Cc: Ingo Molnar <mingo@xxxxxxxxxx>; Juri Lelli
> <juri.lelli@xxxxxxxxxx>;
> > > Vincent Guittot <vincent.guittot@xxxxxxxxxx>; Chen Yu
> > > <yu.c.chen@xxxxxxxxx>; Dietmar Eggemann
> > > <dietmar.eggemann@xxxxxxx>; Steven Rostedt
> <rostedt@xxxxxxxxxxx>;
> > > Ben Segall <bsegall@xxxxxxxxxx>; Mel Gorman <mgorman@xxxxxxx>;
> > > Valentin Schneider <vschneid@xxxxxxxxxx>; K Prateek Nayak
> > > <kprateek.nayak@xxxxxxx>; Shrikanth Hegde
> <sshegde@xxxxxxxxxxxxx>;
> > > Phil Auld <pauld@xxxxxxxxxx>; Andrew Morton
> > > <akpm@xxxxxxxxxxxxxxxxxxxx>; David Hildenbrand
> <david@xxxxxxxxxx>;
> > > linux-kernel@xxxxxxxxxxxxxxx; linux-mm@xxxxxxxxx;
> > > jianyong.wu@xxxxxxxxxxx; Yuan Zhong <zhongyuan@xxxxxxxx>;
> Huangsj
> > > <huangsj@xxxxxxxx>; Fengyu Wang <wangfengyu@xxxxxxxx>; Zhiwei
> Ying
> > > <yingzhiwei@xxxxxxxx>; justin.he@xxxxxxx
> > > Subject: Re: [RFC PATCH v2 02/23] sched/topology: Introduce a NUMA
> > > distance matrix with unique distance values
> > >
> > > On Mon, 2026-09-28 at 09:39 +0000, Jianyong Wu wrote:
> > > > Hi Tim,
> > > >
> > > >
> > > > Makes sense. So, what about the following solution?
> > > > Given a node affinity sequence, a move from src to dst improves the
> > > > affinity of every task whose preferred node i ranks dst better than
> src.
> > > The score
> > > > is then
> > > >
> > > > Di = raw_dist(src, i) - raw_dist(dst, i)
> > > > affinity_bias_i = position of src minus position of dst in node i's
> affinity
> > > >              sequence, counted among the nodes at the same
> > > distance from
> > > >              i (zero when Di is not zero)
> > >
> > > Do we really need an affinity bias?  I think your intention is to use it
> for
> > > breaking a tie.
> > > If there is a tie in affinity score (without injecting bias),
> > > just use the position diff to break the tie.  Having a bias distorts
> > > the affinity score.
> > >
> > >
> > I think there is a misunderstanding about the purpose of the bias.
> > It is not intended to break ties between affinity scores. I want the
> > score itself to reflect opportunities for aggregation, including moves
> > that leave the raw NUMA distance unchanged but follow the fixed node
> > affinity order.
> >
> > That said, I agree with your concern about using artificial node
> > distances in the score calculation. The magnitude of an artificial
> > distance or rank difference should not determine the weight of those
> > aggregation opportunities.
> >
> 
> Thanks a lot for working on this.
> 
> I think there are two level of aggregation:
> LLCs aggregation, and Node aggregation.
> 
> Your current proposal uses a best-effort strategy for multi-LLC/node
> aggregation. It tries to migrate towards a preferred LLC mask, and the
> size of the LLC/node mask is the "range" you calculated in
> task_cache_work().
> 
> Take the LLC case, for example. Suppose there are 6 LLCs, and in
> task_cache_work(), the range is calculated as 3 for process P, which means
> that at most 3 LLCs can hold all the threads of P.
> 
> The strategy allows load balancing to move into the preferred LLC
> mask/range
> in random order:
> 
> suppose the util of each LLCs are:
> LLC0: 45% LLC1: 40%  LLC2: 30%
> 
>    migrate task from LLC4 to LLC1? OK
>    migrate task from LLC4 to LLC0? OK
>    migrate task from LLC2 to LLC0? OK
>    migrate task from LLC0 to LLC1? no
>    ...
> 
> My understanding is that the introduction of distance comparison is
> because the NUMA nodes are not symmetric to each other, so you need a
> realistic sequence to saturate the nodes. However, this might not be
> true for LLCs - they are symmetric, and you can start with a pref CPU
> and get the next LLC via the LLC id sequence. That is to say,
> dist(LLC1, LLC2) = (rank1 + rank2) % k + 1 is not mandatory. Why not
> just generate the preferred_mask by adding the "range" siblings of the
> preferred LLC, and use preferred_mask directly during load balance?
> That seems to be straightforward. So we can get rid of the distance
> comparison (at least for LLCs).
> 

v3 no longer introduces artificial distance values, at either the node or
the LLC level. Ties are now broken with triangular numbers, which
simplifies the ordering.

Thanks
Jianyong

> thanks,
> Chenyu