Hi there,
I think this qualifies as a bug but let me know if this could be a
configuration or hardware issue.
I've been having problems with memory leaks when I run programs on
large--up to 250MB text files. (I know this is huge, but that's the 3
billion base human genome for you.) At first I though it was a Perl
problem but I later found that a completely unrelated C program also
caused memory leaks. I recently upgraded to the 2.4 kernel, hoping to
solve these problems (see below). However, the memory leaks are still
happening and this time I know the problem is at a deeper level than my
programs. Some standard UNIX programs are leaking a lot of memory. I
would appreciate some advice and ultimately, a fix. Unfortunately, My
programming skills are not sufficient for tinkering with the kernel
source. Thank you, in advance for your help. Details follow.
Regards,
Craig Cummings
Here are the specs for my system:
Dell Precision XPSt700, Pentium III, 512 MB RAM
I've recently upgraded from Red Hat 6.2 with the 2.2.14 kernel to
Red Hat 7, then built the 2.4.2 kernel on my own.
Here's what happens with grep:
Output of free, freshly booted system:
total used free shared buffers cached
Mem: 513616 47516 466100 0 2476 27048
-/+ buffers/cache: 17992 495624
Swap: 128480 0 128480
Output of free after grep 'NT_005289' Data/hs_chr12.fa:
total used free shared buffers cached
Mem: 513616 183548 330068 0 2624 159616
-/+ buffers/cache: 21308 492308
Swap: 128480 0 128480
Output of grep 'NT_005289' Data/hs_chr2.fa:
>gi|12728771|ref|NT_005289.2|Hs2_5446 Homo sapiens chromosome 2 working draft sequence segment
Output of free after this grep:
total used free shared buffers cached
Mem: 513616 424272 89344 0 2860 394232
-/+ buffers/cache: 27180 486436
Swap: 128480 0 128480
Output of grep 'NT_005289' Data/hs_chr2.fa:
>gi|12728771|ref|NT_005289.2|Hs2_5446 Homo sapiens chromosome 2 working draft sequence segment
Output of free after this grep:
total used free shared buffers cached
Mem: 513616 424272 89344 0 2860 394232
-/+ buffers/cache: 27180 486436
Swap: 128480 0 128480
File sizes of the two files grep'ed:
-rw-rw-r-- 1 cummings genomics 135744469 Mar 12 22:09 Data/hs_chr12.fa
-rw-rw-r-- 1 cummings genomics 240244039 Mar 12 22:24 Data/hs_chr2.fa
Note that these file sizes are equivalent to the amount of memory leaked
when grep is called on that file.
When I grep the same file a second time, very little additional memory is
leaked.
This same phenomenon occurs when I run a different UNIX program, e.g. wc:
Output of wc -l Data/hs_chr3.fa:
2915465 Data/hs_chr3.fa
Output of free:
total used free shared buffers cached
Mem: 513616 511520 2096 0 1252 481020
-/+ buffers/cache: 29248 484368
Swap: 128480 0 128480
Interestingly, after running wc a second time on the same file, it goes
very fast and very little additional memory is leaked:
total used free shared buffers cached
Mem: 513616 510732 2884 0 1204 480948
-/+ buffers/cache: 28580 485036
Swap: 128480 40 128440
-------------------------------------------
Craig Cummings, Ph.D.
Relman Laboratory
Stanford University School of Medicine
Department of Microbiology and Immunology
e-mail: cummings@cmgm.stanford.edu
phone: 650-498-5998
fax: 650-852-3291
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Fri Mar 23 2001 - 21:00:18 EST