Re: [RFC -V5] autonuma: Migrate on fault among multiple bound nodes

[Huang, Ying]

Mel Gorman <mgorman@xxxxxxx> writes:

> On Wed, Nov 18, 2020 at 01:19:52PM +0800, Huang Ying wrote:
>> Now, AutoNUMA can only optimize the page placement among the NUMA
>
> Note that the feature is referred to as NUMA_BALANCING in the kernel
> configs as AUTONUMA as it was first presented was not merged. The sysctl
> for it is kernel.numa_balancing as you already note. So use NUMAB or
> NUMA_BALANCING but not AUTONUMA because at least a new person grepping
> for NUMA_BALANCING or variants will find it where as autonuma only creeped
> into the powerpc arch code.

Sure.  Will change this.

>
> If exposing to userspace, the naming should definitely be consistent.
>
>> - sysctl knob numa_balancing can enable/disable the NUMA balancing
>>   globally.
>> 
>> - even if sysctl numa_balancing is enabled, the NUMA balancing will be
>>   disabled for the memory areas or applications with the explicit memory
>>   policy by default.
>> 
>> - MPOL_F_AUTONUMA can be used to enable the NUMA balancing for the
>>   applications when specifying the explicit memory policy.
>> 
>
> MPOL_F_NUMAB

Sure, will change it to MPOL_F_NUMA_BALANCING.

>> Various page placement optimization based on the NUMA balancing can be
>> done with these flags.  As the first step, in this patch, if the
>> memory of the application is bound to multiple nodes (MPOL_BIND), and
>> in the hint page fault handler the accessing node are in the policy
>> nodemask, the page will be tried to be migrated to the accessing node
>> to reduce the cross-node accessing.
>> 
>
> The patch still lacks supporting data. It really should have a basic
> benchmark of some sort serving as an example of how the policies should
> be set and a before/after comparison showing the throughput of MPOL_BIND
> accesses spanning 2 or more nodes is faster when numa balancing is enabled.

Sure.  Will add some basic benchmark data and usage example.

> A man page update should also be added clearly outlining when an
> application should consider using it with the linux-api people cc'd
> for review.

Yes.  Will Cc linux-api for review and will submit patches to
manpages.git after the API is finalized.

> The main limitation is that if this requires application modification,
> it may never be used. For example, if an application uses openmp places
> that translates into bind then openmp needs knowledge of the flag.
> Similar limitations apply to MPI. This feature has a risk that no one
> uses it.

My plan is to add a new option to `numactl`
(https://github.com/numactl/numactl/), so users who want to enable NUMA
balancing within the constrains of NUMA binding can use that.  I can
reach some Openstack and Kubernate developers to check whether it's
possible to add the support to these software.  For other applications,
Yes, it may take long time for the new flag to be used.

>> Huang Ying (2):
>>   mempolicy: Rename MPOL_F_MORON to MPOL_F_MOPRON
>>   autonuma: Migrate on fault among multiple bound nodes
>> ---
>>  include/uapi/linux/mempolicy.h | 4 +++-
>>  mm/mempolicy.c                 | 9 +++++++++
>>  2 files changed, 12 insertions(+), 1 deletion(-)
>> 
>> diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
>> index 3354774af61e..adb49f13840e 100644
>> --- a/include/uapi/linux/mempolicy.h
>> +++ b/include/uapi/linux/mempolicy.h
>> @@ -28,12 +28,14 @@ enum {
>>  /* Flags for set_mempolicy */
>>  #define MPOL_F_STATIC_NODES	(1 << 15)
>>  #define MPOL_F_RELATIVE_NODES	(1 << 14)
>> +#define MPOL_F_AUTONUMA		(1 << 13) /* Optimize with AutoNUMA if possible */
>>  
>
> Order by flag usage, correct the naming.

I will correct the naming.  Sorry, what does "order" refer to?

>>  /*
>>   * MPOL_MODE_FLAGS is the union of all possible optional mode flags passed to
>>   * either set_mempolicy() or mbind().
>>   */
>> -#define MPOL_MODE_FLAGS	(MPOL_F_STATIC_NODES | MPOL_F_RELATIVE_NODES)
>> +#define MPOL_MODE_FLAGS							\
>> +	(MPOL_F_STATIC_NODES | MPOL_F_RELATIVE_NODES | MPOL_F_AUTONUMA)
>>  
>>  /* Flags for get_mempolicy */
>>  #define MPOL_F_NODE	(1<<0)	/* return next IL mode instead of node mask */
>
> How does an application discover if MPOL_F_NUMAB is supported by the
> current running kernel? It looks like they might receive -EINVAL (didn't
> check for sure).

Yes.

> In that case, a manual page is defintely needed to
> explain that an error can be returned if the flag is used and the kernel
> does not support it so the application can cover by falling back to a
> strict binding. If it fails silently, then that also needs to be documented
> because it'll lead to different behaviour depending on the running
> kernel.

Sure.  Will describe this in the manual page.

Best Regards,
Huang, Ying