Re: [PATCH 2/2] mm: Update NUMA counter threshold size

[Mel Gorman]

On Tue, Aug 15, 2017 at 10:51:21AM -0700, Tim Chen wrote:
> On 08/15/2017 10:30 AM, Mel Gorman wrote:
> > On Tue, Aug 15, 2017 at 09:55:39AM -0700, Tim Chen wrote:
> 
> >>
> >> Doubling the threshold and counter size will help, but not as much
> >> as making them above u8 limit as seen in Kemi's data:
> >>
> >>       125         537         358906028 <==> system by default (base)
> >>       256         468         412397590
> >>       32765       394(-26.6%) 488932078(+36.2%) <==> with this patchset
> >>
> >> For small system making them u8 makes sense.  For larger ones the
> >> frequent local counter overflow into the global counter still
> >> causes a lot of cache bounce.  Kemi can perhaps collect some data
> >> to see what is the gain from making the counters u8. 
> >>
> > 
> > The same comments hold. The increase of a cache line is undesirable but
> > there are other places where the overall cost can be reduced by special
> > casing based on how this counter is used (always incrementing by one).
> 
> Can you be more explicit of what optimization you suggest here and changes
> to inc/dec_zone_page_state?  Seems to me like we will still overflow
> the local counter with the same frequency unless the threshold and
> counter size is changed.

One of the helpers added is __inc_zone_numa_state which doesn't have a
symmetrical __dec_zone_numa_state because the counter is always
incrementing. Because of this, there is little or no motivation to
update the global value by threshold >> 1 because with both inc/dec, you
want to avoid a corner case whereby a loop of inc/dec would do an
overflow every time. Instead, you can always apply the full threshold
and clear it which is fewer operations and halves the frequency at which
the global value needs to be updated.

-- 
Mel Gorman
SUSE Labs