Re: Problem in Page Cache Replacement

[metin d]

Hi Fengguang,

I run tests and attached the results. The line below I guess shows the data-1 page caches.

0x000000080000006cÂÂÂ ÂÂ 6584051ÂÂÂ 25718Â __RU_lA___________________P________ÂÂÂ referenced,uptodate,lru,active,private
Metin


________________________________
From: Jaegeuk Hanse <jaegeuk.hanse@xxxxxxxxx>
To: Fengguang Wu <fengguang.wu@xxxxxxxxx> 
Cc: metin d <metdos@xxxxxxxxx>; Jan Kara <jack@xxxxxxx>; "linux-kernel@xxxxxxxxxxxxxxx" <linux-kernel@xxxxxxxxxxxxxxx>; "linux-mm@xxxxxxxxx" <linux-mm@xxxxxxxxx> 
Sent: Wednesday, November 21, 2012 11:42 AM
Subject: Re: Problem in Page Cache Replacement

On 11/21/2012 05:02 PM, Fengguang Wu wrote:
> On Wed, Nov 21, 2012 at 04:34:40PM +0800, Jaegeuk Hanse wrote:
>> Cc Fengguang Wu.
>>
>> On 11/21/2012 04:13 PM, metin d wrote:
>>>>Â Â Curious. Added linux-mm list to CC to catch more attention. If you run
>>>> echo 1 >/proc/sys/vm/drop_caches does it evict data-1 pages from memory?
>>> I'm guessing it'd evict the entries, but am wondering if we could run any more diagnostics before trying this.
>>>
>>> We regularly use a setup where we have two databases; one gets used frequently and the other one about once a month. It seems like the memory manager keeps unused pages in memory at the expense of frequently used database's performance.
>>> My understanding was that under memory pressure from heavily
>>> accessed pages, unused pages would eventually get evicted. Is there
>>> anything else we can try on this host to understand why this is
>>> happening?
> We may debug it this way.
>
> 1) run 'fadvise data-2 0 0 dontneed' to drop data-2 cached pages
>Â ÂÂÂ(please double check via /proc/vmstat whether it does the expected work)
>
> 2) run 'page-types -r' with root, to view the page status for the
>Â ÂÂÂremaining pages of data-1
>
> The fadvise tool comes from Andrew Morton's ext3-tools. (source code attached)
> Please compile them with options "-Dlinux -I. -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE"
>
> page-types can be found in the kernel source tree tools/vm/page-types.c
>
> Sorry that sounds a bit twisted.. I do have a patch to directly dump
> page cache status of a user specified file, however it's not
> upstreamed yet.

Hi Fengguang,

Thanks for you detail steps, I think metin can have a try.

   ÂÂÂflags  page-count  ÂÂÂMB symbolic-flags long-symbolic-flags
0x0000000000000000Â Â Â Â 607699Â ÂÂÂ2373 
___________________________________
0x0000000100000000Â Â Â Â 343227Â ÂÂÂ1340 
_______________________r___________Â Â reserved

But I have some questions of the print of page-type:

Is 2373MB here mean total memory in used include page cache? I don't 
think so.
Which kind of pages will be marked reserved?
Which line of long-symbolic-flags is for page cache?

Regards,
Jaegeuk

>
> Thanks,
> Fengguang
>
>>> On Tue 20-11-12 09:42:42, metin d wrote:
>>>> I have two PostgreSQL databases named data-1 and data-2 that sit on the
>>>> same machine. Both databases keep 40 GB of data, and the total memory
>>>> available on the machine is 68GB.
>>>>
>>>> I started data-1 and data-2, and ran several queries to go over all their
>>>> data. Then, I shut down data-1 and kept issuing queries against data-2.
>>>> For some reason, the OS still holds on to large parts of data-1's pages
>>>> in its page cache, and reserves about 35 GB of RAM to data-2's files. As
>>>> a result, my queries on data-2 keep hitting disk.
>>>>
>>>> I'm checking page cache usage with fincore. When I run a table scan query
>>>> against data-2, I see that data-2's pages get evicted and put back into
>>>> the cache in a round-robin manner. Nothing happens to data-1's pages,
>>>> although they haven't been touched for days.
>>>>
>>>> Does anybody know why data-1's pages aren't evicted from the page cache?
>>>> I'm open to all kind of suggestions you think it might relate to problem.
>>>Â Â Curious. Added linux-mm list to CC to catch more attention. If you run
>>> echo 1 >/proc/sys/vm/drop_caches
>>>Â Â does it evict data-1 pages from memory?
>>>
>>>> This is an EC2 m2.4xlarge instance on Amazon with 68 GB of RAM and no
>>>> swap space. The kernel version is:
>>>>
>>>> $ uname -r
>>>> 3.2.28-45.62.amzn1.x86_64
>>>> Edit:
>>>>
>>>> and it seems that I use one NUMA instance, if you think that it can a problem.
>>>>
>>>> $ numactl --hardware
>>>> available: 1 nodes (0)
>>>> node 0 cpus: 0 1 2 3 4 5 6 7
>>>> node 0 size: 70007 MB
>>>> node 0 free: 360 MB
>>>> node distances:
>>>> nodeÂÂÂ0
>>>>Â ÂÂÂ0:Â 10             flags	page-count       MB  symbolic-flags			long-symbolic-flags
0x0000000000000000	   5508317    21516  ___________________________________	
0x0000000100000000	    335993     1312  _______________________r___________	reserved
0x0000002100000000	     35634      139  _______________________r____O______	reserved,owner_private
0x0000000000010000	     45069      176  ________________T__________________	compound_tail
0x0000002000000000	      1516        5  ____________________________O______	owner_private
0x0000000800000004	         1        0  __R_______________________P________	referenced,private
0x0000000000008000	        10        0  _______________H___________________	compound_head
0x0000000000000004	         1        0  __R________________________________	referenced
0x0000000800000024	       166        0  __R__l____________________P________	referenced,lru,private
0x0000000400000028	       295        1  ___U_l___________________d_________	uptodate,lru,mappedtodisk
0x0001000400000028	         3        0  ___U_l___________________d_____I___	uptodate,lru,mappedtodisk,readahead
0x0000000000000028	         1        0  ___U_l_____________________________	uptodate,lru
0x000000040000002c	    262144     1024  __RU_l___________________d_________	referenced,uptodate,lru,mappedtodisk
0x000000080000002c	         5        0  __RU_l____________________P________	referenced,uptodate,lru,private
0x000000000000403c	       185        0  __RUDl________b____________________	referenced,uptodate,dirty,lru,swapbacked
0x0000000800000060	       163        0  _____lA___________________P________	lru,active,private
0x0000000800000064	     36739      143  __R__lA___________________P________	referenced,lru,active,private
0x0000000400000068	    527810     2061  ___U_lA__________________d_________	uptodate,lru,active,mappedtodisk
0x0000000800000068	       576        2  ___U_lA___________________P________	uptodate,lru,active,private
0x0000000c00000068	       116        0  ___U_lA__________________dP________	uptodate,lru,active,mappedtodisk,private
0x000000080000006c	   6584051    25718  __RU_lA___________________P________	referenced,uptodate,lru,active,private
0x000000040000006c	   1302211     5086  __RU_lA__________________d_________	referenced,uptodate,lru,active,mappedtodisk
0x0000000c0000006c	       431        1  __RU_lA__________________dP________	referenced,uptodate,lru,active,mappedtodisk,private
0x000000000000006c	       128        0  __RU_lA____________________________	referenced,uptodate,lru,active
0x0000000800000074	         2        0  __R_DlA___________________P________	referenced,dirty,lru,active,private
0x0000000000004078	        56        0  ___UDlA_______b____________________	uptodate,dirty,lru,active,swapbacked
0x000000000000407c	       122        0  __RUDlA_______b____________________	referenced,uptodate,dirty,lru,active,swapbacked
0x000000080000007c	         1        0  __RUDlA___________________P________	referenced,uptodate,dirty,lru,active,private
0x0000000000008080	     14495       56  _______S_______H___________________	slab,compound_head
0x0000000000000080	    250498      978  _______S___________________________	slab
0x0000000000000400	   2990908    11683  __________B________________________	buddy
0x0000000000000800	        16        0  ___________M_______________________	mmap
0x0000000100000804	         1        0  __R________M___________r___________	referenced,mmap,reserved
0x000000060004082c	       391        1  __RU_l_____M______u_____md_________	referenced,uptodate,lru,mmap,unevictable,mlocked,mappedtodisk
0x0000000a0004082c	       321        1  __RU_l_____M______u_____m_P________	referenced,uptodate,lru,mmap,unevictable,mlocked,private
0x0000000000004838	      8450       33  ___UDl_____M__b____________________	uptodate,dirty,lru,mmap,swapbacked
0x000000000000483c	      2045        7  __RUDl_____M__b____________________	referenced,uptodate,dirty,lru,mmap,swapbacked
0x0000000800000868	        19        0  ___U_lA____M______________P________	uptodate,lru,active,mmap,private
0x0000000400000868	         5        0  ___U_lA____M_____________d_________	uptodate,lru,active,mmap,mappedtodisk
0x000000040000086c	      1891        7  __RU_lA____M_____________d_________	referenced,uptodate,lru,active,mmap,mappedtodisk
0x000000080000086c	       126        0  __RU_lA____M______________P________	referenced,uptodate,lru,active,mmap,private
0x0000000000004878	        85        0  ___UDlA____M__b____________________	uptodate,dirty,lru,active,mmap,swapbacked
0x000000000000487c	      2263        8  __RUDlA____M__b____________________	referenced,uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000005008	        13        0  ___U________a_b____________________	uptodate,anonymous,swapbacked
0x0000000000005808	        16        0  ___U_______Ma_b____________________	uptodate,mmap,anonymous,swapbacked
0x0000000200045828	         8        0  ___U_l_____Ma_b___u_____m__________	uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x000000020004582c	       651        2  __RU_l_____Ma_b___u_____m__________	referenced,uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x0000000000005868	      8058       31  ___U_lA____Ma_b____________________	uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c	        42        0  __RU_lA____Ma_b____________________	referenced,uptodate,lru,active,mmap,anonymous,swapbacked
             total	  17922048    70008

             flags	page-count       MB  symbolic-flags			long-symbolic-flags
0x0000000000000000	    121628      475  ___________________________________	
0x0000000100000000	    335993     1312  _______________________r___________	reserved
0x0000002100000000	     35634      139  _______________________r____O______	reserved,owner_private
0x0000000000010000	     45429      177  ________________T__________________	compound_tail
0x0000002000000000	      1389        5  ____________________________O______	owner_private
0x0000000400000001	         6        0  L________________________d_________	locked,mappedtodisk
0x0000000000008000	        10        0  _______________H___________________	compound_head
0x0000000000000004	         1        0  __R________________________________	referenced
0x0000000400000021	        64        0  L____l___________________d_________	locked,lru,mappedtodisk
0x0001000400000021	         1        0  L____l___________________d_____I___	locked,lru,mappedtodisk,readahead
0x0000000800000024	       171        0  __R__l____________________P________	referenced,lru,private
0x0000000400000028	      4093       15  ___U_l___________________d_________	uptodate,lru,mappedtodisk
0x0001000400000028	        59        0  ___U_l___________________d_____I___	uptodate,lru,mappedtodisk,readahead
0x0000000000000028	         1        0  ___U_l_____________________________	uptodate,lru
0x000000040000002c	   8598032    33586  __RU_l___________________d_________	referenced,uptodate,lru,mappedtodisk
0x000000080000002c	        10        0  __RU_l____________________P________	referenced,uptodate,lru,private
0x000000000000403c	       185        0  __RUDl________b____________________	referenced,uptodate,dirty,lru,swapbacked
0x0000000800000060	       163        0  _____lA___________________P________	lru,active,private
0x0000000800000064	     36741      143  __R__lA___________________P________	referenced,lru,active,private
0x0000000400000068	    527834     2061  ___U_lA__________________d_________	uptodate,lru,active,mappedtodisk
0x0000000800000068	       695        2  ___U_lA___________________P________	uptodate,lru,active,private
0x0000000c00000068	       116        0  ___U_lA__________________dP________	uptodate,lru,active,mappedtodisk,private
0x000000080000006c	   6584066    25719  __RU_lA___________________P________	referenced,uptodate,lru,active,private
0x000000040000006c	   1325273     5176  __RU_lA__________________d_________	referenced,uptodate,lru,active,mappedtodisk
0x0000000c0000006c	       431        1  __RU_lA__________________dP________	referenced,uptodate,lru,active,mappedtodisk,private
0x000000000000006c	       128        0  __RU_lA____________________________	referenced,uptodate,lru,active
0x0000000000004078	        56        0  ___UDlA_______b____________________	uptodate,dirty,lru,active,swapbacked
0x000000000000407c	       122        0  __RUDlA_______b____________________	referenced,uptodate,dirty,lru,active,swapbacked
0x000000080000007c	         1        0  __RUDlA___________________P________	referenced,uptodate,dirty,lru,active,private
0x0000000000008080	     14571       56  _______S_______H___________________	slab,compound_head
0x0000000000000080	    250546      978  _______S___________________________	slab
0x0000000000000400	     14701       57  __________B________________________	buddy
0x0000000000000800	        16        0  ___________M_______________________	mmap
0x0000000100000804	         1        0  __R________M___________r___________	referenced,mmap,reserved
0x000000060004082c	       391        1  __RU_l_____M______u_____md_________	referenced,uptodate,lru,mmap,unevictable,mlocked,mappedtodisk
0x0000000a0004082c	       321        1  __RU_l_____M______u_____m_P________	referenced,uptodate,lru,mmap,unevictable,mlocked,private
0x0000000000004838	      8385       32  ___UDl_____M__b____________________	uptodate,dirty,lru,mmap,swapbacked
0x000000000000483c	      2045        7  __RUDl_____M__b____________________	referenced,uptodate,dirty,lru,mmap,swapbacked
0x0000000800000868	        19        0  ___U_lA____M______________P________	uptodate,lru,active,mmap,private
0x0000000400000868	         5        0  ___U_lA____M_____________d_________	uptodate,lru,active,mmap,mappedtodisk
0x000000040000086c	      1891        7  __RU_lA____M_____________d_________	referenced,uptodate,lru,active,mmap,mappedtodisk
0x000000080000086c	       126        0  __RU_lA____M______________P________	referenced,uptodate,lru,active,mmap,private
0x0000000000004878	        85        0  ___UDlA____M__b____________________	uptodate,dirty,lru,active,mmap,swapbacked
0x000000000000487c	      2263        8  __RUDlA____M__b____________________	referenced,uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000005008	         4        0  ___U________a_b____________________	uptodate,anonymous,swapbacked
0x0000000000005808	        25        0  ___U_______Ma_b____________________	uptodate,mmap,anonymous,swapbacked
0x0000000200045828	         8        0  ___U_l_____Ma_b___u_____m__________	uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x000000020004582c	       651        2  __RU_l_____Ma_b___u_____m__________	referenced,uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x0000000000005868	      7623       29  ___U_lA____Ma_b____________________	uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c	        39        0  __RU_lA____Ma_b____________________	referenced,uptodate,lru,active,mmap,anonymous,swapbacked
             total	  17922048    70008