Re: Q: memory management

Richard Gooch (Richard.Gooch@atnf.CSIRO.AU)
Tue, 2 Jun 1998 11:46:52 +1000


Peter Horton writes:
>
> I have a machine with 72MB of RAM and 128MB of swap.
>
> If I try to do an allocation of 512MB using sbrk(), it fails. However, I
> can successfully do 512 sbrk()'s of 1MB. This causes a problem if I store
> the pointers to the 1MB blocks, and then try and write to the 512MB of
> memory I have just allocated ... the kernel seems to lock up solid. If I
> write to each 1MB as it is allocated, the allocation fails around 180MB
> (amount of free memory).
>
> If I understand correctly I can allocate more memory than I have because
> the physical memory pages are not allocated until you access a block
> (allowing sparse data structures).

Yep.

> So, my first question is why does the single sbrk(512MB) fail whilst the
> 512 x sbrk(1MB) succeeds ?

It's a similar dumb-arse bug that you'll find with mips4_IRIX6.2
machines. On my Linux box I can allocate up to 2046 MBytes, but the
largest single block is only 128 MBytes. This box has 128 MBytes of
RAM and 128 MBytes of swap.

On a certain mips4_IRIX6.2 machine (with 1 GByte RAM and 10 GBytes of
swap) I can malloc 6454 MBytes but the single largest block I can
allocate is only 2048 MBytes. This is if I allocate ever increasing
blocks. If the first thing my programme does is allocate 800 MBytes,
that will fail.
Allocating 800 MBytes by mmapping /dev/zero works straight away.

Clearly, the Linux and IRIX implementations of sbrk(2) are flawed. I
note that Solaris 2 doesn't have these problems.

> Secondly, does the kernel lock up, or did I just not wait long enough for
> it to come back ? ... I'm not keen to try it again, as it resulted in an
> hour of fsck'ing ... 8-)

You could easily spend hours waiting for your machine to stop
thrashing. Heavy paging also wears down your discs. The solution is:
Don't Do That[tm].

If you don't have sparse data, then don't use swap, just buy more
RAM. This prevents thrashing, since you can't over-commit VM.

If you are stuck with large sparse arrays larger than RAM, be careful
not to touch more pages than you have available.
You can improve things by putting your swap parition on a separate
disc. Swap should *never* be on the same disc as your data and should
preferably not be on the disc with your OS and programmes.

Regards,

Richard....

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu