"User does not understand how *NIX systems utilize memory" is
not the same thing as "bug". What you are seeing is the kernel
using memory for file system cache.
On 03/22/2001 12:58 -0800, Craig Cummings wrote:
>> Hi there,
>>
>> I think this qualifies as a bug but let me know if this could be a
>> configuration or hardware issue.
>>
>> I've been having problems with memory leaks when I run programs on
>> large--up to 250MB text files. (I know this is huge, but that's the 3
>> billion base human genome for you.) At first I though it was a Perl
>> problem but I later found that a completely unrelated C program also
>> caused memory leaks. I recently upgraded to the 2.4 kernel, hoping to
>> solve these problems (see below). However, the memory leaks are still
>> happening and this time I know the problem is at a deeper level than my
>> programs. Some standard UNIX programs are leaking a lot of memory. I
>> would appreciate some advice and ultimately, a fix. Unfortunately, My
>> programming skills are not sufficient for tinkering with the kernel
>> source. Thank you, in advance for your help. Details follow.
>>
>> Regards,
>>
>> Craig Cummings
>>
>>
>> Here are the specs for my system:
>> Dell Precision XPSt700, Pentium III, 512 MB RAM
>> I've recently upgraded from Red Hat 6.2 with the 2.2.14 kernel to
>> Red Hat 7, then built the 2.4.2 kernel on my own.
>>
>> Here's what happens with grep:
>>
>> Output of free, freshly booted system:
>>
>> total used free shared buffers cached
>> Mem: 513616 47516 466100 0 2476 27048
>> -/+ buffers/cache: 17992 495624
>> Swap: 128480 0 128480
>>
>> Output of free after grep 'NT_005289' Data/hs_chr12.fa:
>>
>> total used free shared buffers cached
>> Mem: 513616 183548 330068 0 2624 159616
>> -/+ buffers/cache: 21308 492308
>> Swap: 128480 0 128480
>>
>> Output of grep 'NT_005289' Data/hs_chr2.fa:
>>
>> >gi|12728771|ref|NT_005289.2|Hs2_5446 Homo sapiens chromosome 2 working draft sequence segment
>>
>> Output of free after this grep:
>>
>> total used free shared buffers cached
>> Mem: 513616 424272 89344 0 2860 394232
>> -/+ buffers/cache: 27180 486436
>> Swap: 128480 0 128480
>>
>> Output of grep 'NT_005289' Data/hs_chr2.fa:
>>
>> >gi|12728771|ref|NT_005289.2|Hs2_5446 Homo sapiens chromosome 2 working draft sequence segment
>>
>> Output of free after this grep:
>>
>> total used free shared buffers cached
>> Mem: 513616 424272 89344 0 2860 394232
>> -/+ buffers/cache: 27180 486436
>> Swap: 128480 0 128480
>>
>> File sizes of the two files grep'ed:
>>
>> -rw-rw-r-- 1 cummings genomics 135744469 Mar 12 22:09 Data/hs_chr12.fa
>> -rw-rw-r-- 1 cummings genomics 240244039 Mar 12 22:24 Data/hs_chr2.fa
>>
>> Note that these file sizes are equivalent to the amount of memory leaked
>> when grep is called on that file.
>>
>> When I grep the same file a second time, very little additional memory is
>> leaked.
>>
>>
>> This same phenomenon occurs when I run a different UNIX program, e.g. wc:
>>
>> Output of wc -l Data/hs_chr3.fa:
>>
>> 2915465 Data/hs_chr3.fa
>>
>> Output of free:
>>
>> total used free shared buffers cached
>> Mem: 513616 511520 2096 0 1252 481020
>> -/+ buffers/cache: 29248 484368
>> Swap: 128480 0 128480
>>
>> Interestingly, after running wc a second time on the same file, it goes
>> very fast and very little additional memory is leaked:
>>
>> total used free shared buffers cached
>> Mem: 513616 510732 2884 0 1204 480948
>> -/+ buffers/cache: 28580 485036
>> Swap: 128480 40 128440
>>
>>
>> -------------------------------------------
>> Craig Cummings, Ph.D.
>>
>> Relman Laboratory
>> Stanford University School of Medicine
>> Department of Microbiology and Immunology
>>
>> e-mail: cummings@cmgm.stanford.edu
>> phone: 650-498-5998
>> fax: 650-852-3291
>>
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
End of included message
-- +--------------------------+------------------------------+ | Tim Walberg | tewalberg@mediaone.net | | 828 Marshall Ct. | www.concentric.net/~twalberg | | Palatine, IL 60074 | | +--------------------------+------------------------------+
This archive was generated by hypermail 2b29 : Fri Mar 23 2001 - 21:00:18 EST