Re: performance regression due to commit e82e0561("mm: vmscan: obey proportional scanning requirements for kswapd")

From: Yuanhan Liu
Date: Tue Mar 18 2014 - 02:37:24 EST


On Sat, Mar 15, 2014 at 08:56:10PM -0700, Hugh Dickins wrote:
> On Fri, 14 Mar 2014, Mel Gorman wrote:
> > On Thu, Mar 13, 2014 at 05:44:57AM -0700, Hugh Dickins wrote:
> > > On Wed, 12 Mar 2014, Mel Gorman wrote:
> > > > On Tue, Feb 18, 2014 at 04:01:22PM +0800, Yuanhan Liu wrote:
... snip ...

> > > I missed Yuanhan's mail, but seeing your reply reminds me of another
> > > issue with that proportionality patch - or perhaps more thought would
> > > show them to be two sides of the same issue, with just one fix required.
> > > Let me throw our patch into the cauldron.
> > >
> > > [PATCH] mm: revisit shrink_lruvec's attempt at proportionality
> > >
> > > We have a memcg reclaim test which exerts a certain amount of pressure,
> > > and expects to see a certain range of page reclaim in response. It's a
> > > very wide range allowed, but the test repeatably failed on v3.11 onwards,
> > > because reclaim goes wild and frees up almost everything.
> > >
> > > This wild behaviour bisects to Mel's "scan_adjusted" commit e82e0561dae9
> > > "mm: vmscan: obey proportional scanning requirements for kswapd". That
> > > attempts to achieve proportionality between anon and file lrus: to the
> > > extent that once one of those is empty, it then tries to empty the other.
> > > Stop that.
> > >
> > > Signed-off-by: Hugh Dickins <hughd@xxxxxxxxxx>
> > > ---
> > >
> > > We've been running happily with this for months; but all that time it's
> > > been on my TODO list with a "needs more thought" tag before we could
> > > upstream it, and I never got around to that. We also have a somewhat
> > > similar, but older and quite independent, fix to get_scan_count() from
> > > Suleiman, which I'd meant to send along at the same time: I'll dig that
> > > one out tomorrow or the day after.
>
> I've sent that one out now in a new thread
> https://lkml.org/lkml/2014/3/15/168
> and also let's tie these together with Hannes's
> https://lkml.org/lkml/2014/3/14/277
>
> > >
> >
> > I ran a battery of page reclaim related tests against it on top of
> > 3.14-rc6. Workloads showed small improvements in their absolute performance
> > but actual IO behaviour looked much better in some tests. This is the
> > iostats summary for the test that showed the biggest different -- dd of
> > a large file on ext3.
> >
> > 3.14.0-rc6 3.14.0-rc6
> > vanilla proportional-v1r1
> > Mean sda-avgqz 1045.64 224.18
> > Mean sda-await 2120.12 506.77
> > Mean sda-r_await 18.61 19.78
> > Mean sda-w_await 11089.60 2126.35
> > Max sda-avgqz 2294.39 787.13
> > Max sda-await 7074.79 2371.67
> > Max sda-r_await 503.00 414.00
> > Max sda-w_await 35721.93 7249.84
> >
> > Not all workloads benefitted. The same workload on ext4 showed no useful
> > difference. btrfs looks like
> >
> > 3.14.0-rc6 3.14.0-rc6
> > vanilla proportional-v1r1
> > Mean sda-avgqz 762.69 650.39
> > Mean sda-await 2438.46 2495.15
> > Mean sda-r_await 44.18 47.20
> > Mean sda-w_await 6109.19 5139.86
> > Max sda-avgqz 2203.50 1870.78
> > Max sda-await 7098.26 6847.21
> > Max sda-r_await 63.02 156.00
> > Max sda-w_await 19921.70 11085.13
> >
> > Better but not as dramatically so. I didn't analyse why. A workload that
> > had a large anonymous mapping with large amounts of IO in the background
> > did not show any regressions so based on that and the fact the patch looks
> > ok, here goes nothing;
> >
> > Acked-by: Mel Gorman <mgorman@xxxxxxx>
>
> Big thank you, Mel, for doing so much work on it, and so very quickly.
> I get quite lost in the numbers myself: I'm much more convinced of it
> by your numbers and ack.
>
> >
> > You say it's already been tested for months but it would be nice if the
> > workload that generated this thread was also tested.
>
> Yes indeed: Yuanhan, do you have time to try this patch for your
> testcase? I'm hoping it will prove at least as effective as your
> own suggested patch, but please let us know what you find - thanks.

Hi Hugh,

Sure, and sorry to tell you that this patch introduced another half
performance descrease from avg 60 MB/s to 30 MB/s in this testcase.

Moreover, the dd throughput for each process was steady before, however,
it's quite bumpy from 20 MB/s to 40 MB/s w/ this patch applied, and thus
got a avg of 30 MB/s:

11327188992 bytes (11 GB) copied, 300.014 s, 37.8 MB/s
1809373+0 records in
1809372+0 records out
7411187712 bytes (7.4 GB) copied, 300.008 s, 24.7 MB/s
3068285+0 records in
3068284+0 records out
12567691264 bytes (13 GB) copied, 300.001 s, 41.9 MB/s
1883877+0 records in
1883876+0 records out
7716356096 bytes (7.7 GB) copied, 300.002 s, 25.7 MB/s
1807674+0 records in
1807673+0 records out
7404228608 bytes (7.4 GB) copied, 300.024 s, 24.7 MB/s
1796473+0 records in
1796472+0 records out
7358349312 bytes (7.4 GB) copied, 300.008 s, 24.5 MB/s
1905655+0 records in
1905654+0 records out
7805558784 bytes (7.8 GB) copied, 300.016 s, 26.0 MB/s
2819168+0 records in
2819167+0 records out
11547308032 bytes (12 GB) copied, 300.025 s, 38.5 MB/s
1848381+0 records in
1848380+0 records out
7570964480 bytes (7.6 GB) copied, 300.005 s, 25.2 MB/s
3023133+0 records in
3023132+0 records out
12382748672 bytes (12 GB) copied, 300.024 s, 41.3 MB/s
1714585+0 records in
1714584+0 records out
7022936064 bytes (7.0 GB) copied, 300.011 s, 23.4 MB/s
1835132+0 records in
1835131+0 records out
7516696576 bytes (7.5 GB) copied, 299.998 s, 25.1 MB/s
1733341+0 records in


--yliu
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/