(no subject)

From: leechin@ucsb.edu
Date: Tue Feb 08 2000 - 17:26:54 EST


Sorry for taking too much of your time on this, but I really appreciate your help.

PS: If this is not a discussion for the kernel mailing list, please let me know and I will take it off line :)

> >lots of trivial-sized allocations are a really bad idea in any case.
> >doing it in 1M chunks will give you more like 3G.
>
> But what if I have an application that doesent know how much memory it will
> need?
> then allocating lots is a horrendous design flaw.

> My application is reading in a file, parsing it and constructing a
> huge in-memory database.

why not just mmap the file?
or at least malloc big chunks, which is an obviously smart thing
to do in the first place.

** I cant do this since the format of the file is different from the structure I need to put in memory... This is a flat file that I need to put into a tree like datastructure. I need to add one node per entry in the file at a time. So the algorithm blindly mallocs for every record it finds. To understand you correctly, you are suggesting that I have my own memory management code that mallocs from the OS even bigger chunks and gives smaller chunks to the tree algorithm, correct? In any event, its clear you are suggesting I go away from malloc and use my own memory management code + mmap. Correct?

>> Should I write my own library that will first mmap
>> a large (say 2 GB) memory and then dish out smaller chunks to the
>> application from there on an as need babsis?

> sure. though 2G makes no sense, since malloc is page-granular,
> and you'll see no advantage over sensible chunk sizes (say, a > few MB
> at a time.)

> This is what malloc should
> really do.

> no, it's not. malloc is not really intended to be used for this
> kind of long-term allocation. sure, it usually works, but it can't
> please all possible uses, and the most time critical uses are small
> blocks that get used and freed, not large chunks that get used throughout the
> life of the app.

>> What is the general approach taken for these kind of
>> applications?

> malloc your own big chunks. or better yet, mmap the database.
>all serious DB's do the latter, afaik.

> > the issue is
> > actually that the kernel normally permits 3G of VM to each task,
> > but starts mmaped allocations at the 1G mark. that limits the sbrk
>
> It seems complicated for an application that wants to use a large amount of
> memory to know that it first has to use brk then switch to mmap after a
> 1GB... or am I understanding this incorrectly? It also might break
> portability, right?

> no. first, an app which uses a large amount of memory normally
> does so in nontrivial allocations. those will get put into > mmaps.
> next, malloc cannot possibly serve all applications perfectly.
> finally, >GB apps are still rare.

> > heap, of course. but it's easy and not harmfull to shift the
> > starting point for mmap if you must use tiny mallocs. a better
> How do I do this? Is this a kernel change?

> the simplest tweak is to change TASK_UNMAPPED_BASE in >asm/processor.h
>from TASK_SIZE/3 to 2*TASK_SIZE/3 (for 2G heap). actually, I >found
>that you can change it to TASK_SIZE-512M if you don't have any >big mmaps.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Tue Feb 15 2000 - 21:00:13 EST