RE: [PATCH] cpu-topology: warn if NUMA configurations conflicts with lower layer

[Zengtao (B)]

> -----Original Message-----
> From: Valentin Schneider [mailto:valentin.schneider@xxxxxxx]
> Sent: Thursday, January 02, 2020 9:22 PM
> To: Zengtao (B); Sudeep Holla
> Cc: Linuxarm; Greg Kroah-Hartman; Rafael J. Wysocki;
> linux-kernel@xxxxxxxxxxxxxxx; Morten Rasmussen
> Subject: Re: [PATCH] cpu-topology: warn if NUMA configurations conflicts
> with lower layer
> 
> On 02/01/2020 12:47, Zengtao (B) wrote:
> >>
> >> As I said, wrong configurations need to be detected when generating
> >> DT/ACPI if possible. The above will print warning on systems with
> NUMA
> >> within package.
> >>
> >> NUMA:  0-7, 8-15
> >> core_siblings:   0-15
> >>
> >> The above is the example where the die has 16 CPUs and 2 NUMA
> nodes
> >> within a package, your change throws error to the above config which is
> >> wrong.
> >>
> > From your example, the core 7 and core 8 has got different LLC but the
> same Low
> > Level cache?
> 
> AFAIA what matters here is memory controllers, less so LLCs. Cores within
> a single die could have private LLCs and separate memory controllers, or
> shared LLC and separate memory controllers.
> 
> > From schedule view of point, lower level sched domain should be a subset
> of higher
> > Level sched domain.
> >
> 
> Right, and that is checked when you have sched_debug on the cmdline
> (or write 1 to /sys/kernel/debug/sched_debug & regenerate the sched
> domains)
> 

No, here I think you don't get my issue, please try to understand my example
First:.

*************************************
NUMA:         0-2,  3-7
core_siblings:    0-3,  4-7
*************************************
When we are building the sched domain, per the current code:
(1) For core 3
 MC sched domain fallbacks to 3~7
 DIE sched domain is 3~7
(2) For core 4:
 MC sched domain is 4~7
 DIE sched domain is 3~7

When we are build sched groups for the MC level:
(1). core3's sched groups chain is built like as: 3->4->5->6->7->3 
(2). core4's sched groups chain is built like as: 4->5->6->7->4 
so after (2), 
core3's sched groups is overlapped, and it's not a chain any more.
In the afterwards usecase of core3's sched groups, deadloop happens.

And it's difficult for the scheduler to find out such errors,
that is why I think a warning is necessary here.

> Now, I don't know how this plays out for the numa-in-package topologies
> like
> the one suggested by Sudeep. x86 and AMD had to play some games to
> get
> numa-in-package topologies working, see e.g.
> 
>   cebf15eb09a2 ("x86, sched: Add new topology for multi-NUMA-node
> CPUs")
> 
> perhaps you need to "lie" here and ensure sub-NUMA sched domain levels
> are
> a subset of the NUMA levels, i.e. lie for core_siblings. I might go and
> play with this to see how it behaves.