Re: [lkp-robot] [mm, vmscan] 5e56dfbd83: fsmark.files_per_sec -11.1% regression

[Mel Gorman]

On Thu, Feb 23, 2017 at 08:35:45AM +0100, Michal Hocko wrote:
> >      57.60 ±  0%      -11.1%      51.20 ±  0%  fsmark.files_per_sec
> >     607.84 ±  0%       +9.0%     662.24 ±  1%  fsmark.time.elapsed_time
> >     607.84 ±  0%       +9.0%     662.24 ±  1%  fsmark.time.elapsed_time.max
> >      14317 ±  6%      -12.2%      12568 ±  7%  fsmark.time.involuntary_context_switches
> >       1864 ±  0%       +0.5%       1873 ±  0%  fsmark.time.maximum_resident_set_size
> >      12425 ±  0%      +23.3%      15320 ±  3%  fsmark.time.minor_page_faults
> >      33.00 ±  3%      -33.9%      21.80 ±  1%  fsmark.time.percent_of_cpu_this_job_got
> >     203.49 ±  3%      -28.1%     146.31 ±  1%  fsmark.time.system_time
> >     605701 ±  0%       +3.6%     627486 ±  0%  fsmark.time.voluntary_context_switches
> >     307106 ±  2%      +20.2%     368992 ±  9%  interrupts.CAL:Function_call_interrupts
> >     183040 ±  0%      +23.2%     225559 ±  3%  softirqs.BLOCK
> >      12203 ± 57%     +236.4%      41056 ±103%  softirqs.NET_RX
> >     186118 ±  0%      +21.9%     226922 ±  2%  softirqs.TASKLET
> >      14317 ±  6%      -12.2%      12568 ±  7%  time.involuntary_context_switches
> >      12425 ±  0%      +23.3%      15320 ±  3%  time.minor_page_faults
> >      33.00 ±  3%      -33.9%      21.80 ±  1%  time.percent_of_cpu_this_job_got
> >     203.49 ±  3%      -28.1%     146.31 ±  1%  time.system_time
> >       3.47 ±  3%      -13.0%       3.02 ±  1%  turbostat.%Busy
> >      99.60 ±  1%       -9.6%      90.00 ±  1%  turbostat.Avg_MHz
> >      78.69 ±  1%       +1.7%      80.01 ±  0%  turbostat.CorWatt
> >       3.56 ± 61%      -91.7%       0.30 ± 76%  turbostat.Pkg%pc2
> >     207790 ±  0%       -8.2%     190654 ±  1%  vmstat.io.bo
> >   30667691 ±  0%      +65.9%   50890669 ±  1%  vmstat.memory.cache
> >   34549892 ±  0%      -58.4%   14378939 ±  4%  vmstat.memory.free
> >       6768 ±  0%       -1.3%       6681 ±  1%  vmstat.system.cs
> >  1.089e+10 ±  2%      +13.4%  1.236e+10 ±  3%  cpuidle.C1E-IVT.time
> >   11475304 ±  2%      +13.4%   13007849 ±  3%  cpuidle.C1E-IVT.usage
> >    2.7e+09 ±  6%      +13.2%  3.057e+09 ±  3%  cpuidle.C3-IVT.time
> >    2954294 ±  6%      +14.3%    3375966 ±  3%  cpuidle.C3-IVT.usage
> >   96963295 ± 14%      +17.5%  1.139e+08 ± 12%  cpuidle.POLL.time
> >       8761 ±  7%      +17.6%      10299 ±  9%  cpuidle.POLL.usage
> >   30454483 ±  0%      +66.4%   50666102 ±  1%  meminfo.Cached
> > 
> > Do you see what's happening?
> 
> not really. All I could see in the previous data was that the memory
> locality was different (and better) with my patch, which I cannot
> explain either because get_scan_count is always per-node thing. Moreover
> the change shouldn't make any difference for normal GFP_KERNEL requests
> on 64b systems because the reclaim index covers all zones so there is
> nothing to skip over.
> 
> > Or is there anything we can do to improve fsmark benchmark setup to
> > make it more reasonable?
> 
> Unfortunatelly I am not an expert on this benchmark. Maybe Mel knows
> better.

There is not much to be an expert on with that benchmark. It creates a
bunch of files of the requested size for a number of iterations. In async
configurations, it can be heavily skewed by the first few iterations until
dirty limits are hit. Once that point is reached, the files/sec drops
rapidly to some value below the writing speed of the underlying device.
Hence, looking at the average performance of it is risky and very sensitive
to exact timing unless this is properly accounted for.

In async configurations, stalls are dominated by balance_dirty_pages
and some filesystem details such as whether it needs to wait for space
in a transaction log. That also limits the overall performance of the
workload. Once the stable phase is reached, there still will be quite some
variability due to the timing of the writeback threads that cause a bit
of jitter as well as the usual concerns with multiple threads writing to
different parts of the disk.

When NUMA is taken into account, it is important to consider the size of
the NUMA nodes as assymetric sizes will affect when remote memory is used
and to a lesser extent when balance_dirty_pages is triggered.

The benchmark is what it is. You can force it to generate stable figures
but it won't have the same behaviour so it all depends on how you define
"reasonable".

At the very minimum, take into account that an average of multiple iterations
will be skewed early in the workloads lifetime by the fact it hasn't hit
dirty limits yet.

-- 
Mel Gorman
SUSE Labs