Re: Memory Management
From: Márcio Oliveira
Date: Fri Jul 22 2005 - 18:31:31 EST
Roger Heflin wrote:
I have seen RH3.0 crash on 32GB systems because it has too
much memory tied up in write cache. It required update 2
(this was a while ago) and a change of a parameter in /proc
to prevent the crash, it was because of a overagressive
write caching change RH implemented in the kernel resulted
in the crash. This crash was an OOM related crash. To
duplicate the bug, you booted the machine and ran a dd
to create a very large file filling the disk.
We did test and did determine that it did not appear to have
the issue if you had less than 28GB of ram, this was on an
itanium machine, so I don't know if it occurs on other arches,
and if it occurs at the same memory limits on the other arches
either.
Roger
-----Original Message-----
From: linux-kernel-owner@xxxxxxxxxxxxxxx
[mailto:linux-kernel-owner@xxxxxxxxxxxxxxx] On Behalf Of
Márcio Oliveira
Sent: Friday, July 22, 2005 2:42 PM
To: Neil Horman
Cc: arjanv@xxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx
Subject: Re: Memory Management
Neil Horman wrote:
On Fri, Jul 22, 2005 at 11:32:52AM -0300, Márcio Oliveira wrote:
Neil Horman wrote:
On Thu, Jul 21, 2005 at 10:40:54AM -0300, Márcio Oliveira wrote:
http://people.redhat.com/nhorman/papers/rhel3_vm.pdf
I wrote this with norm awhile back. It may help you out.
Regards
Neil
Neil,
Thanks.~10-12GB of total RAM (16GB) are
How can Proc virtual memory parameters like
inactive_clean_percent,
overcommit_memory, overcommit_ratio and page_cache help
me to solve
/ reduce Out Of Memory conditions on servers with 16GB
RAM and lots
of GB swap?
I wouldn't touch memory overcommit if you are already
seeing out of
memory issues. If you are using lots of pagecache, I
would suggest
increasing inactive_clean percent, reducing the
pagecahce.max value,
and modifying the bdflush parameters in the above document
such that
bdflush runs sooner, more often, and does more work per
iteration.
This will help you move data in pagecache back to disk more
aggressively so that memory will be available for other purposes,
like heap allocations. Also if you're using a Red Hat
kernel and you
have 16GB of ram in your system, you're a good candidate for the
hugemem kernel. Rather than a straightforward out of memory
condition, you may be seeing a exhaustion of your kernels address
space (check LowFree in /proc/meminfo). In this even the hugemem
kernel will help you in that it increases your Low Memory address
space from 1GB to 4GB, preventing some OOM conditions.
Kernel does not free cached memory (~10-12GB of total RAM
- 16GB). Is
there some way to force the kernel to free cached memory?
Cached memory is freed on demand. Just because its listed
under the
cached line
below doesn't mean it can't be freed and used for another
purpose.
Implement
the tunings above, and your situation should improve.
Regards
Neil
/proc/meminfo:
total: used: free: shared: buffers: cached:
Mem: 16603488256 16523333632 80154624 0
70651904 13194563584
Swap: 17174257664 11771904 17162485760
MemTotal: 16214344 kB
MemFree: 78276 kB
Buffers: 68996 kB
Cached: 12874808 kB
Thanks to all.
Marcio.
Neil,
Thanks for the answers!
The following lines are the Out Of Memory log:
Jul 20 13:45:44 server kernel: Out of Memory: Killed
process 23716 (oracle).
Jul 20 13:45:44 server kernel: Fixed up OOM kill of mm-less task
Jul 20 13:45:45 server su(pam_unix)[3848]: session closed
for user root
Jul 20 13:45:48 server kernel: Mem-info:
Jul 20 13:45:48 server kernel: Zone:DMA freepages: 1884 min: 0
low: 0 high: 0
Jul 20 13:45:48 server kernel: Zone:Normal freepages: 1084
min: 1279
low: 4544 high: 6304
Jul 20 13:45:48 server kernel: Zone:HighMem
freepages:386679 min: 255
low: 61952 high: 92928
Jul 20 13:45:48 server kernel: Free pages: 389647
(386679 HighMem)
Jul 20 13:45:48 server kernel: ( Active: 2259787/488777,
inactive_laundry: 244282, inactive_clean: 244366, free: 389647 )
Jul 20 13:45:48 server kernel: aa:0 ac:0 id:0 il:0 ic:0 fr:1884
Jul 20 13:45:48 server kernel: aa:1620 ac:1801 id:231
il:15 ic:0 fr:1085
Jul 20 13:45:48 server kernel: aa:1099230 ac:1157136 id:488536
il:244277 ic:244366 fr:386679
Jul 20 13:45:48 server kernel: 0*4kB 0*8kB 1*16kB 1*32kB
1*64kB 0*128kB
1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7536kB)Jul 20 13:45:48
server kernel: 55*4kB 9*8kB 19*16kB 9*32kB 0*64kB 1*128kB 1*256kB
0*512kB 1*1024kB 1*2048kB 0*4096kB = 4340kB)
Jul 20 13:45:48 server kernel: 291229*4kB 46179*8kB 711*16kB 1*32kB
1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB =
1546716kB)
Jul 20 13:45:48 server kernel: Swap cache: add 192990,
delete 189665,
find 21145/90719, race 0+0
Jul 20 13:45:48 server kernel: 139345 pages of slabcache
Jul 20 13:45:48 server kernel: 1890 pages of kernel stacks
Jul 20 13:45:48 server kernel: 0 lowmem pagetables, 274854 highmem
pagetables
Jul 20 13:45:48 server kernel: Free swap: 16749720kB
Jul 20 13:45:49 server kernel: 4194304 pages of RAM
Jul 20 13:45:49 server kernel: 3899360 pages of HIGHMEM
Jul 20 13:45:49 server kernel: 140718 reserved pages
Jul 20 13:45:49 server kernel: 35350398 pages shared
Jul 20 13:45:49 server kernel: 3325 pages swap cached
/proc/meminfo LowFree info:
LowFree: 17068 kB ------> Do you think this
value is too low?
No that should be plenty of lowFree, but that number can
change quickly
depending on workload.
Zone:Normal freepages: 1084 min: 1279 low: 4544 high:
6304 ---->
(freepages < min) It's normal?
Zone:HighMem freepages:386679 min: 255 low: 61952 high:
92928 ---->
(freepages < min) It's normal?
You're beneath your low water mark in the normal (lowmem)
zone for free pages,
so your kernel is likely trying to get lots of data moved to
disk. Although
given that you're largest buddy list has a 2048K chunk free,
I'm hard pressed to
see how you aren't able to get memory when you need it. Do
you have a module
loaded in your kernel that might require such large memory
allocations.
Neil
Thanks a lot Neil!
Márcio Oliveira.
Neil,
Thanks for the help.
I have a storage attached to the server. Maybe the storage module
require lots of memory.
Maybe the "LowFree" be wrong (out of OOM time), so there
is possible
that "LowFree" value be too small on the OOM condition.
Is there a way to identify if the Low Memory is too small? (some
program, command, daemon...)
The server has 16GB RAM and 16GB swap. When the OOM kill conditions
happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free
swap. Is that indicate that the server can't allocate Low Memory and
starts OOM conditions? Because the High Memory is OK, right?
Thanks again!
Márcio.
-
To unsubscribe from this list: send the line "unsubscribe
linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Roger, thanks for the information.
I'm using Update 4 kernels (2.4.21-27.ELsmp - This kernel have some
mm / oom fixes) and don't have big problems when create large files,
plus the server is a 32-bit machine.
Neil said that the problem can be Low Memory and I think it too.
I read the following message on the list:
http://marc.theaimsgroup.com/?l=linux-kernel&m=112044530919567&w=4
The problem seems like a I/O issue. Can I/O (like storage devices)
consume a large amount of low memory? Neil also said that the kernel is
trying to move lots of data to the disk and it's a module might require
such large memory. Somebody know how can I identify what is using Low
Memory on my system?
The older questions in the message are:
The server has 16GB RAM and 16GB swap. When the OOM kill conditions happens, the system has ~6GB RAM used, ~10GB RAM cached and 16GB free swap. Is that indicate that the server can't allocate Low Memory and starts OOM conditions? Because the High Memory is OK, right?
Thanks to all!
Regards,
Márcio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/