Re: [RFC PATCH v1 13/13] mm: splice local lists onto the front of the LRU

[Daniel Jordan]

On 02/01/2018 06:30 PM, Tim Chen wrote:
> On 01/31/2018 03:04 PM, daniel.m.jordan@xxxxxxxxxx wrote:
> > Now that release_pages is scaling better with concurrent removals from
> > the LRU, the performance results (included below) showed increased
> > contention on lru_lock in the add-to-LRU path.
> >
> > To alleviate some of this contention, do more work outside the LRU lock.
> > Prepare a local list of pages to be spliced onto the front of the LRU,
> > including setting PageLRU in each page, before taking lru_lock.  Since
> > other threads use this page flag in certain checks outside lru_lock,
> > ensure each page's LRU links have been properly initialized before
> > setting the flag, and use memory barriers accordingly.
> >
> > Performance Results
> >
> > This is a will-it-scale run of page_fault1 using 4 different kernels.
> >
> >              kernel     kern #
> >
> >            4.15-rc2          1
> >    large-zone-batch          2
> >       lru-lock-base          3
> >     lru-lock-splice          4
> >
> > Each kernel builds on the last.  The first is a baseline, the second
> > makes zone->lock more scalable by increasing an order-0 per-cpu
> > pagelist's 'batch' and 'high' values to 310 and 1860 respectively
> > (courtesy of Aaron Lu's patch), the third scales lru_lock without
> > splicing pages (the previous patch in this series), and the fourth adds
> > page splicing (this patch).
> >
> > N tasks mmap, fault, and munmap anonymous pages in a loop until the test
> > time has elapsed.
> >
> > The process case generally does better than the thread case most likely
> > because of mmap_sem acting as a bottleneck.  There's ongoing work
> > upstream[*] to scale this lock, however, and once it goes in, my
> > hypothesis is the thread numbers here will improve.

Neglected to mention my hardware:
  2-socket system, 44 cores, 503G memory, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

> > kern #  ntask     proc      thr        proc    stdev         thr    stdev
> >                 speedup  speedup       pgf/s                pgf/s
> >       1      1                       705,533    1,644     705,227    1,122
> >       2      1     2.5%     2.8%     722,912      453     724,807      728
> >       3      1     2.6%     2.6%     724,215      653     723,213      941
> >       4      1     2.3%     2.8%     721,746      272     724,944      728
> >
> > kern #  ntask     proc      thr        proc    stdev         thr    stdev
> >                 speedup  speedup       pgf/s                pgf/s
> >       1      4                     2,525,487    7,428   1,973,616   12,568
> >       2      4     2.6%     7.6%   2,590,699    6,968   2,123,570   10,350
> >       3      4     2.3%     4.4%   2,584,668   12,833   2,059,822   10,748
> >       4      4     4.7%     5.2%   2,643,251   13,297   2,076,808    9,506
> >
> > kern #  ntask     proc      thr        proc    stdev         thr    stdev
> >                 speedup  speedup       pgf/s                pgf/s
> >       1     16                     6,444,656   20,528   3,226,356   32,874
> >       2     16     1.9%    10.4%   6,566,846   20,803   3,560,437   64,019
> >       3     16    18.3%     6.8%   7,624,749   58,497   3,447,109   67,734
> >       4     16    28.2%     2.5%   8,264,125   31,677   3,306,679   69,443
> >
> > kern #  ntask     proc      thr        proc    stdev         thr    stdev
> >                 speedup  speedup       pgf/s                pgf/s
> >       1     32                    11,564,988   32,211   2,456,507   38,898
> >       2     32     1.8%     1.5%  11,777,119   45,418   2,494,064   27,964
> >       3     32    16.1%    -2.7%  13,426,746   94,057   2,389,934   40,186
> >       4     32    26.2%     1.2%  14,593,745   28,121   2,486,059   42,004
> >
> > kern #  ntask     proc      thr        proc    stdev         thr    stdev
> >                 speedup  speedup       pgf/s                pgf/s
> >       1     64                    12,080,629   33,676   2,443,043   61,973
> >       2     64     3.9%     9.9%  12,551,136  206,202   2,684,632   69,483
> >       3     64    15.0%    -3.8%  13,892,933  351,657   2,351,232   67,875
> >       4     64    21.9%     1.8%  14,728,765   64,945   2,485,940   66,839
> >
> > [*] https://lwn.net/Articles/724502/  Range reader/writer locks
> >      https://lwn.net/Articles/744188/  Speculative page faults
>
> The speedup looks pretty nice and seems to peak at 16 tasks.  Do you have an explanation of what
> causes the drop from 28.2% to 21.9% going from 16 to 64 tasks?

The system I was testing on had 44 cores, so part of the decrease in % speedup is just saturating the hardware (e.g. memory bandwidth).  At 64 processes, we start having to share cores.  Page faults per second did continue to increase each time we added more processes, though, so there's no anti-scaling going on.

> Was
> the loss in performance due to increased contention on LRU lock when more tasks running
> results in a higher likelihood of hitting the sentinel?

That seems to be another factor, yes.  I used lock_stat to measure it, and it showed that wait time on lru_lock nearly tripled when going from 32 to 64 processes, but I also take lock_stat with a grain of salt as it changes the timing/interaction between processes.

> If I understand
> your patchset correctly, you will need to acquire LRU lock for sentinel page. Perhaps an increase
> in batch size could help?

Actually, I did try doing that.  In this series the batch size is PAGEVEC_SIZE (14).  When I did a run with PAGEVEC_SIZE*4, the performance stayed nearly the same for all but the 64 process case, where it dropped by ~10%.  One explanation is as a process runs through one batch, it holds the batch lock longer before it has to switch batches, creating more opportunity for contention.


By the way, we're also working on another approach to scaling this look:
    https://marc.info/?l=linux-mm&m=151746028405581

We plan to implement that idea and see how it compares performance-wise and diffstat-wise with this.