Re: [PATCH 1/6] sched/topology: introduce for_each_numa_hop_node() / sched_numa_hop_node()

[Yury Norov]

On Wed, Dec 18, 2024 at 06:04:53AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Wed, Dec 18, 2024 at 11:23:40AM +0100, Andrea Righi wrote:
> ...
> > > So, this would work but given that there is nothing dynamic about this
> > > ordering, would it make more sense to build the ordering and store it
> > > per-node? Then, the iteration just becomes walking that array.
> > 
> > I've also considered doing that. I don't know if it'd work with offline
> > nodes, but maybe we can just check node_online(node) at each iteration and
> > skip those that are not online.

for_each_numa_hop_mask() only traverses N_CPU nodes, and N_CPU nodes have
proper distances.

I think that for_each_numa_hop_node() should match for_each_numa_hop_mask().
It would be good to cross-test them to ensure that they generate the same
order at least for N_CPU nodes.

If you think that for_each_numa_hop_node() should traverse non-N_CPU nodes,
you need a 'node_state' parameter. This will allow to make sure that at
least N_CPU portion works correctly.

> Yeah, there can be e.g. for_each_possible_node_by_dist() wheke nodes with
> unknown distances (offline ones?) are put at the end and then there's also
> for_each_online_node_by_dist() which filters out offline ones, and the
> ordering can be updated from a CPU hotplug callback.

We can assign UINT_MAX for those nodes I guess?

> The ordering can be
> probably put in an rcu protected array? I'm not sure what's the
> synchronization convention around node on/offlining. Is that protected
> together with CPU on/offlining?

The machinery is already there, we just need another array of nodemasks - 
sched_domains_numa_nodes in addition to sched_domains_numa_nodes. The
last one is already protected by RCU, and we need to update new array every
time when sched_domains_numa_nodes updated.
 
> Given that there usually aren't that many nodes, the current implementation
> is probably fine too, so please feel free to ignore this suggestion for now
> too.

I agree. The number of nodes on typical system is 1 or 2. Even if
it's 8, the Andrea's bubble sort will be still acceptable. So, I'm
OK with O(N^2) if you guys OK with it. I only would like to have
this choice explained in commit message.

Thanks,
Yury