[PATCH 2.6.17.3] Memory Management: High-Memory Scalability Issue

[Satapathy, Soumendu Sekhar]

From: Soumendu Sekhar Satapathy <Soumendu.Satapathy@xxxxxxxxxxxxx>


top - 07:04:46 up  5:49,  3 users,  load average: 4.85, 4.77, 4.66
Mem:  65883624k total, 65721200k used,   162424k free,     6100k buffers
Swap:  1049576k total,  1049576k used,        0k free,   666144k cached

The test case to reproduce the issue, eats up most of the memory and performs disk I/O. Given above is the "top" o/p which denotes the state of the system.

When this test case is run in a low RAM system ( less than 4G ),  the disk I/O rate is very much acceptable and the test case completes in a normal time but in a high RAM system of more than 32G, we get low I/O rates and the completion time of the test case is much greater than the completion time of a low RAM machine. 

It could be a linux scalability issue with High-Memory. When the memory is more with application data,it takes more time for the test case to complete.


File : mm/vmscan.c and routine: shrink_active_list( )

Swap tendency is calculated as follows as per version 2.6.17.3.
       swap_tendency = mapped_ratio / 2 + distress +  vm_swappiness ;
       
When the swap space is zero lets say, the total exercise of scanning pages to reclaim goes in vain since there are no free slots available hence the  swap tendency should also depend on the number of free slot available in the swap device. 

swap tendency = A + B + C , 

where A = mapped_ratio / 2, B = distress 

and C = ( vm_swappiness * nr_swap_pages ) / 32

This third quantity C, is calculated by multiplying vm_swappiness with a ratio  of nr_swap_pages  divided by 32. The static quantity 32, is the number of pages reclaimed per invocation and  nr_swap_pages is the number 
of free slots available in the swap device. When the swap device have more free slots,  the swap tendency should proportionately increase. At the same time when the free slots in the swap device approaches zero the swap tendency should proportionately decrease

These analysis has been use to modify the current formula and the result is as follows....

swap_tendency = mapped_ratio / 2 + distress + ( vm_swappiness * nr_swap_pages ) / 32 ;


The test case took 6 hours to complete in a 4G machine and the I/O rates were preety fast but the same test case took more than 8 hours to complete in version 2.6.17.3 in a 64G system  and the I/O rates were preety slower as compared to the former. After the patch was applied, the test case completed in a time quite close to a 4G RAM system i.e 6 hours and the I/O rates were also much better. 

The following are the Test Results:-
############################################################################

Kernel version Tested : 2.6.17.3
System Memory:           64G
State: Without patch
Time Taken for Completion of the Test Case : 11 Hours approx.
The following are the I/O rates

Doing 50 times: /usr/bin/bonnie -s 1024 -m M1135- -v 3 ... 

              ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-    Iteration
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU    Start Date&Time

M1135- 3*1024 23779 95.2 20222 95.3 17248 54.7 26313 84.6 28849 58.9  343.9  3.2
M1135- 3*1024 18897 94.1 12698 83.1 11408 72.1 27800 84.7 35135 53.2  329.9  3.6
M1135- 3*1024 16945 87.1 12787 91.9 14282 59.6 24895 79.6 27435 62.6  339.3  3.4
############################################################################

Kernel version Tested :   2.6.17.3
System Memory:           64G
State:   Patch was applied
Time Taken for Completion of the Test Case :       6 Hours and 30 minutes.
The following are the I/O rates

Doing 50 times: /usr/bin/bonnie -s 1024 -m M1135- -v 3 ... 

              ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-    Iteration
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU    Start Date&Time

M1135- 3*1024 25746 93.9 44001 93.6 25134 29.5 40780 81.9 63560 24.0  388.0  2.4
M1135- 3*1024 25655 92.7 39142 91.2 24959 28.6 40366 81.3 61830 27.0  356.6  3.2
M1135- 3*1024 28122 93.0 30958 97.7 23702 33.2 41839 84.2 66231 24.3  388.8  3.1
############################################################################

Kernel version Tested :   2.6.17.3
System Memory:           4G
State:   No Patch was applied
Time Taken for Completion of the Test Case : 6 Hours approx.

############################################################################


Signed-off-by: Soumendu Sekhar Satapathy <Soumendu.Satapathy@xxxxxxxxxxxxx>

Attachment: mem_scal.patch
Description: mem_scal.patch