Re: [PATCH v3] sched/topology: Improve load balancing on AMD EPYC

[Peter Zijlstra]

On Tue, Jul 23, 2019 at 12:42:48PM +0100, Mel Gorman wrote:
> On Tue, Jul 23, 2019 at 11:48:30AM +0100, Matt Fleming wrote:
> > SD_BALANCE_{FORK,EXEC} and SD_WAKE_AFFINE are stripped in sd_init()
> > for any sched domains with a NUMA distance greater than 2 hops
> > (RECLAIM_DISTANCE). The idea being that it's expensive to balance
> > across domains that far apart.
> > 
> > However, as is rather unfortunately explained in
> > 
> >   commit 32e45ff43eaf ("mm: increase RECLAIM_DISTANCE to 30")
> > 
> > the value for RECLAIM_DISTANCE is based on node distance tables from
> > 2011-era hardware.
> > 
> > Current AMD EPYC machines have the following NUMA node distances:
> > 
> > node distances:
> > node   0   1   2   3   4   5   6   7
> >   0:  10  16  16  16  32  32  32  32
> >   1:  16  10  16  16  32  32  32  32
> >   2:  16  16  10  16  32  32  32  32
> >   3:  16  16  16  10  32  32  32  32
> >   4:  32  32  32  32  10  16  16  16
> >   5:  32  32  32  32  16  10  16  16
> >   6:  32  32  32  32  16  16  10  16
> >   7:  32  32  32  32  16  16  16  10
> > 
> > where 2 hops is 32.
> > 
> > The result is that the scheduler fails to load balance properly across
> > NUMA nodes on different sockets -- 2 hops apart.
> > 
> > For example, pinning 16 busy threads to NUMA nodes 0 (CPUs 0-7) and 4
> > (CPUs 32-39) like so,
> > 
> >   $ numactl -C 0-7,32-39 ./spinner 16
> > 
> > causes all threads to fork and remain on node 0 until the active
> > balancer kicks in after a few seconds and forcibly moves some threads
> > to node 4.
> > 
> > Override node_reclaim_distance for AMD Zen.
> > 
> > Signed-off-by: Matt Fleming <matt@xxxxxxxxxxxxxxxxxxx>
> > Cc: "Suthikulpanit, Suravee" <Suravee.Suthikulpanit@xxxxxxx>
> > Cc: Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx>
> > Cc: "Lendacky, Thomas" <Thomas.Lendacky@xxxxxxx>
> > Cc: Borislav Petkov <bp@xxxxxxxxx>
> 
> Acked-by: Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx>
> 
> The only caveat I can think of is that a future generation of Zen might
> take a different magic number than 32 as their remote distance. If or
> when this happens, it'll need additional smarts but lacking a crystal
> ball, we can cross that bridge when we come to it.

I just suggested to Matt on IRC we could do something along these lines,
but we can do that later.

--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1611,6 +1611,12 @@ void sched_init_numa(void)
 	}
 
 	/*
+	 * Set the reclaim distance at 2 hops instead of at a fixed distance value.
+	 */
+	if (level >= 2)
+		node_reclaim_distance = sched_domains_numa_distance[2];
+
+	/*
 	 * 'level' contains the number of unique distances
 	 *
 	 * The sched_domains_numa_distance[] array includes the actual distance