Re: [PATCH v5 00/10] workingset protection/detection on the anonymous LRU list

[Joonsoo Kim]

2020ë 6ì 3ì (ì) ìí 12:57, Suren Baghdasaryan <surenb@xxxxxxxxxx>ëì ìì:
>
> On Wed, Apr 8, 2020 at 5:50 PM Joonsoo Kim <js1304@xxxxxxxxx> wrote:
> >
> > 2020ë 4ì 9ì (ë) ìì 1:55, Vlastimil Babka <vbabka@xxxxxxx>ëì ìì:
> > >
> > > On 4/3/20 7:40 AM, js1304@xxxxxxxxx wrote:
> > > > From: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx>
> > > >
> > > > Hello,
> > > >
> > > > This patchset implements workingset protection and detection on
> > > > the anonymous LRU list.
> > >
> > > Hi!
> >
> > Hi!
> >
> > > > I did another test to show the performance effect of this patchset.
> > > >
> > > > - ebizzy (with modified random function)
> > > > ebizzy is the test program that main thread allocates lots of memory and
> > > > child threads access them randomly during the given times. Swap-in/out
> > > > will happen if allocated memory is larger than the system memory.
> > > >
> > > > The random function that represents the zipf distribution is used to
> > > > make hot/cold memory. Hot/cold ratio is controlled by the parameter. If
> > > > the parameter is high, hot memory is accessed much larger than cold one.
> > > > If the parameter is low, the number of access on each memory would be
> > > > similar. I uses various parameters in order to show the effect of
> > > > patchset on various hot/cold ratio workload.
> > > >
> > > > My test setup is a virtual machine with 8 cpus and 1024MB memory.
> > > >
> > > > Result format is as following.
> > > >
> > > > Parameter 0.1 ... 1.3
> > > > Allocated memory size
> > > > Throughput for base (larger is better)
> > > > Throughput for patchset (larger is better)
> > > > Improvement (larger is better)
> > > >
> > > >
> > > > * single thread
> > > >
> > > >      0.1      0.3      0.5      0.7      0.9      1.1      1.3
> > > > <512M>
> > > >   7009.0   7372.0   7774.0   8523.0   9569.0  10724.0  11936.0
> > > >   6973.0   7342.0   7745.0   8576.0   9441.0  10730.0  12033.0
> > > >    -0.01     -0.0     -0.0     0.01    -0.01      0.0     0.01
> > > > <768M>
> > > >    915.0   1039.0   1275.0   1687.0   2328.0   3486.0   5445.0
> > > >    920.0   1037.0   1238.0   1689.0   2384.0   3638.0   5381.0
> > > >     0.01     -0.0    -0.03      0.0     0.02     0.04    -0.01
> > > > <1024M>
> > > >    425.0    471.0    539.0    753.0   1183.0   2130.0   3839.0
> > > >    414.0    468.0    553.0    770.0   1242.0   2187.0   3932.0
> > > >    -0.03    -0.01     0.03     0.02     0.05     0.03     0.02
> > > > <1280M>
> > > >    320.0    346.0    410.0    556.0    871.0   1654.0   3298.0
> > > >    316.0    346.0    411.0    550.0    892.0   1652.0   3293.0
> > > >    -0.01      0.0      0.0    -0.01     0.02     -0.0     -0.0
> > > > <1536M>
> > > >    273.0    290.0    341.0    458.0    733.0   1381.0   2925.0
> > > >    271.0    293.0    344.0    462.0    740.0   1398.0   2969.0
> > > >    -0.01     0.01     0.01     0.01     0.01     0.01     0.02
> > > > <2048M>
> > > >     77.0     79.0     95.0    147.0    276.0    690.0   1816.0
> > > >     91.0     94.0    115.0    170.0    321.0    770.0   2018.0
> > > >     0.18     0.19     0.21     0.16     0.16     0.12     0.11
> > > >
> > > >
> > > > * multi thread (8)
> > > >
> > > >      0.1      0.3      0.5      0.7      0.9      1.1      1.3
> > > > <512M>
> > > >  29083.0  29648.0  30145.0  31668.0  33964.0  38414.0  43707.0
> > > >  29238.0  29701.0  30301.0  31328.0  33809.0  37991.0  43667.0
> > > >     0.01      0.0     0.01    -0.01     -0.0    -0.01     -0.0
> > > > <768M>
> > > >   3332.0   3699.0   4673.0   5830.0   8307.0  12969.0  17665.0
> > > >   3579.0   3992.0   4432.0   6111.0   8699.0  12604.0  18061.0
> > > >     0.07     0.08    -0.05     0.05     0.05    -0.03     0.02
> > > > <1024M>
> > > >   1921.0   2141.0   2484.0   3296.0   5391.0   8227.0  14574.0
> > > >   1989.0   2155.0   2609.0   3565.0   5463.0   8170.0  15642.0
> > > >     0.04     0.01     0.05     0.08     0.01    -0.01     0.07
> > > > <1280M>
> > > >   1524.0   1625.0   1931.0   2581.0   4155.0   6959.0  12443.0
> > > >   1560.0   1707.0   2016.0   2714.0   4262.0   7518.0  13910.0
> > > >     0.02     0.05     0.04     0.05     0.03     0.08     0.12
> > > > <1536M>
> > > >   1303.0   1399.0   1550.0   2137.0   3469.0   6712.0  12944.0
> > > >   1356.0   1465.0   1701.0   2237.0   3583.0   6830.0  13580.0
> > > >     0.04     0.05      0.1     0.05     0.03     0.02     0.05
> > > > <2048M>
> > > >    172.0    184.0    215.0    289.0    514.0   1318.0   4153.0
> > > >    175.0    190.0    225.0    329.0    606.0   1585.0   5170.0
> > > >     0.02     0.03     0.05     0.14     0.18      0.2     0.24
> > > >
> > > > As we can see, as allocated memory grows, patched kernel get the better
> > > > result. Maximum improvement is 21% for the single thread test and
> > > > 24% for the multi thread test.
> > >
> > > So, these results seem to be identical since v1. After the various changes up to
> > > v5, should  the benchmark be redone? And was that with a full patchset or
> > > patches 1+2?
> >
> > It was done with a full patchset. I think that these results would not
> > be changed
> > even on v5 since it is improvement from the concept of this patchset and
> > implementation detail doesn't much matter. However, I will redo.
> >
> > > > * EXPERIMENT
> > > > I made a test program to imitates above scenario and confirmed that
> > > > problem exists. Then, I checked that this patchset fixes it.
> > > >
> > > > My test setup is a virtual machine with 8 cpus and 6100MB memory. But,
> > > > the amount of the memory that the test program can use is about 280 MB.
> > > > This is because the system uses large ram-backed swap and large ramdisk
> > > > to capture the trace.
> > > >
> > > > Test scenario is like as below.
> > > >
> > > > 1. allocate cold memory (512MB)
> > > > 2. allocate hot-1 memory (96MB)
> > > > 3. activate hot-1 memory (96MB)
> > > > 4. allocate another hot-2 memory (96MB)
> > > > 5. access cold memory (128MB)
> > > > 6. access hot-2 memory (96MB)
> > > > 7. repeat 5, 6
> > > >
> > > > Since hot-1 memory (96MB) is on the active list, the inactive list can
> > > > contains roughly 190MB pages. hot-2 memory's re-access interval
> > > > (96+128 MB) is more 190MB, so it cannot be promoted without workingset
> > > > detection and swap-in/out happens repeatedly. With this patchset,
> > > > workingset detection works and promotion happens. Therefore, swap-in/out
> > > > occurs less.
> > > >
> > > > Here is the result. (average of 5 runs)
> > > >
> > > > type swap-in swap-out
> > > > base 863240 989945
> > > > patch 681565 809273
> > > >
> > > > As we can see, patched kernel do less swap-in/out.
> > >
> > > Same comment, also v1 has this note:
> >
> > I had tested this scenario on every version of the patchset and found the same
> > trend.
> >
> > > > Note that, this result is gotten from v5.1. Although workingset detection
> > > > works on v5.1, it doesn't work well on v5.5. It looks like that recent
> > > > code change on workingset.c is the reason of this problem. I will
> > > > track it soon.
> > > What was the problem then, assuming it's fixed? Maybe I just missed it
> > > mentioned. Can results now be gathered on 5.6?
> >
> > It was fixed on v2. Change log on v2 "fix a critical bug that uses out of index
> > lru list in workingset_refault()" is for this problem. I should note
> > that clearly.
> >
> > > In general, this patchset seems to be doing the right thing. I haven't reviewed
> > > the code yet, but hope to do so soon. But inevitably, with any changes in this
> > > area there will be workloads that will suffer instead of benefit. That can be
> > > because we are actually doing a wrong thing, or there's a bug in the code, or
> > > the workloads happen to benefit from the current behavior even if it's not the
> > > generally optimal one. And I'm afraid only testing on a variety of workloads can
> > > show that. You mentioned somewhere that your production workloads benefit? Can
> > > it be quantified more? Could e.g. Johannes test this a bit at Facebook, or
> >
> > I cannot share the detail of the test for my production (smart TV)
> > workload. Roughly,
> > it is repeat of various action and app (channel change, volume change,
> > youtube, etc.)
> > on smart TV and it is memory stress test. Result after the workload is:
> >
> > base
> > pswpin 328211
> > pswpout 304015
> >
> > patched
> > pswpin 261884
> > pswpout 276062
> >
> > So, improvement on pswpin and pswpout is roughly 20% and 9%, respectively.
> >
> > > it be quantified more? Could e.g. Johannes test this a bit at Facebook, or
> > > somebody at Google?
> >
> > It's really helpful if someone else could test this on their workload.
>
> Please let me know when the new version (after Johannes' memcg
> charging changes) is available for testing. I'll try them on Android
> workload.
> Thanks.

I have a new version after Johannes' memcg charging changes but now
MM tree has another changes on this area ("mm: balance LRU lists based on
relative thrashing v2") so I need to rebase on it. I guess that at
least two weaks
are required.

Thanks.