Re: [PATCH 4/4] mm: page_alloc: Reduce cost of the fair zone allocation policy

[Mel Gorman]

On Mon, Jun 30, 2014 at 02:14:04PM -0700, Andrew Morton wrote:
> On Mon, 30 Jun 2014 17:48:03 +0100 Mel Gorman <mgorman@xxxxxxx> wrote:
> 
> > The fair zone allocation policy round-robins allocations between zones
> > within a node to avoid age inversion problems during reclaim. If the
> > first allocation fails, the batch counts is reset and a second attempt
> > made before entering the slow path.
> > 
> > One assumption made with this scheme is that batches expire at roughly the
> > same time and the resets each time are justified. This assumption does not
> > hold when zones reach their low watermark as the batches will be consumed
> > at uneven rates.  Allocation failure due to watermark depletion result in
> > additional zonelist scans for the reset and another watermark check before
> > hitting the slowpath.
> > 
> > This patch makes a number of changes that should reduce the overall cost
> > 
> > o Abort the fair zone allocation policy once remote zones are encountered
> > o Use a simplier scan when resetting NR_ALLOC_BATCH
> > o Use a simple flag to identify depleted zones instead of accessing a
> >   potentially write-intensive cache line for counters
> > 
> > On UMA machines, the effect on overall performance is marginal. The main
> > impact is on system CPU usage which is small enough on UMA to begin with.
> > This comparison shows the system CPu usage between vanilla, the previous
> > patch and this patch.
> > 
> >           3.16.0-rc2  3.16.0-rc2  3.16.0-rc2
> >              vanilla checklow-v4 fairzone-v4
> > User          390.13      400.85      396.13
> > System        404.41      393.60      389.61
> > Elapsed      5412.45     5166.12     5163.49
> > 
> > There is a small reduction and it appears consistent.
> > 
> > On NUMA machines, the scanning overhead is higher as zones are scanned
> > that are ineligible for use by zone allocation policy. This patch fixes
> > the zone-order zonelist policy and reduces the numbers of zones scanned
> > by the allocator leading to an overall reduction of CPU usage.
> > 
> >           3.16.0-rc2  3.16.0-rc2  3.16.0-rc2
> >              vanilla checklow-v4 fairzone-v4
> > User          744.05      763.26      778.53
> > System      70148.60    49331.48    44905.73
> > Elapsed     28094.08    27476.72    27378.98
> 
> That's a large change in system time.  Does this all include kswapd
> activity?
> 

I don't have a profile to quantify that exactly. It takes 7 hours to
complete a test on that machine in this configuration and it would take
longer with profiling. I was not testing with profiling enabled as that
invalidates performance tests. I'd expect it'd take the guts of two days
to gather full profiles for it and even then it would be masked by remote
access costs and other factors. It'd be worse considering that automatic
NUMA balancing is enabled and I normally test with that turned on.

However, without the kswapd change there are a lot of retries and
reallocations for pages recently reclaimed. For the fairzone patch there
are far fewer scans of unusable zones to find the lower zones. Considering
the number of allocations required there is simply a lot of overhead that
builds up.

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/