> On Tue, 21 Mar 2000, James Sutherland wrote:
>
> > On 21 Mar 2000 13:40:28 +0100, you wrote:
> >
> > The process CAN be told there isn't enough memory to do what it wants.
> > Just signal it (with a signal it can catch.) It can then use that
> > signal to trigger garbage collection (if it's an ML or Java VM, for
> > example), shrink the cache (an music player), reduce spare processes
> > (Apache), whatever.
>
> Interesting. The page fault routine should trigger a signal to the process
> that caused its invocation. And what even makes you hope that that
> process won't page fault again executing the signal handler (pieces of
> code and data that have been much likely paged-out months ago)?
>
> We are not short of memory. We're short of swap. paging-out is
> the problem,
> not paging-in.
This is why it's important to let the applications know about a potential
resource problem early enough. If you let them know too late, they may not
be able to do a thing about it without worsening it.
Most of my applications can do lots of things if resource consumption is a
problem. For example, they can tell the load manager to divert connections
to other machines and let the connections to it close by themselves. Some of
them can free cached data. They can defer memory-intensive operations for a
later time. They can refuse 'expensive' commands from clients.
All of these things will reduce swap consumption eventually, it's just a
matter of finding out early enough.
> > IMO, it's much better to get a signal which means "we're getting short
> > of memory, folks", which can be handled in ONE place, rather than
> > returning 0 as a pointer - which many apps then try to dereference,
> > ending up segfaulting themselves anyway.
>
> "many apps" do not check malloc() return values? I don't call them apps,
> I call them first time students exercises. B-)
> They're are not able to check malloc() return value but they can setup a
> signal handler that triggers runtime a GC() clever enough not to page
> fault itself? I don't follow you.
Well, I can tell you what my apps do. They have two memory allocation
routines, a hard one and a soft one. The soft one can fail, returning NULL,
and it's used for large allocations that correlate to starting tasks that
aren't essential. When the soft allocator is called, the code checks for
NULL and handles it gracefully.
If the hard allocator gets a NULL from malloc, it frees a portion of its
emergency pool and retries. If enough of the emergency pool is freed, it
begins taking steps to reduce memory consumption as a I discussed above. The
hard allocator never fails, though it may take it a while to get you the
memory you asked for. The hard allocator is used for the millions of 'small'
allocations where it's impractical to deal with an allocation failure.
With a properly sized emergency pool and enough strategies for reducing
memory consumption over time, you can avoid the hassle of having to deal
with memory exhaustion at every memory allocation point (which can be nearly
impossible in large applications). However, you can't do much about stack
expansion, can you?
Sophisticated applications need the tools to allow them to deal with memory
(or memory+swap) exhaustion. A side issue is that applications can't tell
when they're hitting swap either, and so can't take steps to reduce memory
consumption to improve their own (and system) performance.
DS
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Mar 23 2000 - 21:00:34 EST