Re: 2.6.29.1: nfsd: page allocation failure - nfsd or kernel problem?

[Michael Tokarev]

David Rientjes wrote:
> On Thu, 18 Jun 2009, Michael Tokarev wrote:
>
> > David Rientjes wrote:
> > > On Thu, 18 Jun 2009, Michael Tokarev wrote:
> > >
> > > > > 	http://bugzilla.kernel.org/show_bug.cgi?id=13518
> > > > Does not look similar.
> > > >
> > > > I repeated the issue here.  The slab which is growing here is buffer_head.
> > > > It's growing slowly -- right now, after ~5 minutes of constant writes over
> > > > nfs, its size is 428423 objects, growing at about 5000 objects/minute
> > > > rate.
> > > > When stopping writing, the cache shrinks slowly back to an acceptable
> > > > size, probably when the data gets actually written to disk.
> > > Not sure if you're referring to the bugzilla entry or Justin's reported
> > > issue.  Justin's issue is actually allocating a skbuff_head_cache slab while
> > > the system is oom.
> > We have the same issue - I replied to Justin's initial email with exactly
> > the same trace as him.  I didn't see your reply up until today, -- the one
> > you're referring to below.
>
> If it's the exact same trace, then the page allocation failure is 
> occurring as the result of slab's growth of the skbuff_head_cache cache, 
> not buffer_head.

See http://lkml.org/lkml/2009/6/16/550 -- second message in this thread
is mine, it shows exactly the same trace.

> So it appears as though the issue you're raising is that buffer_head is 
> consuming far too much memory, which causes the system to be oom when 
> attempting a GFP_ATOMIC allocation for skbuff_head_cache and is otherwise 
> unseen with alloc_buffer_head() because it is allowed to invoke direct 
> reclaim:
>
> 	$ grep -r alloc_buffer_head\( fs/*
> 	fs/buffer.c:		bh = alloc_buffer_head(GFP_NOFS);
> 	fs/buffer.c:struct buffer_head *alloc_buffer_head(gfp_t gfp_flags)
> 	fs/gfs2/log.c:	bh = alloc_buffer_head(GFP_NOFS | __GFP_NOFAIL);
> 	fs/jbd/journal.c:	new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);
> 	fs/jbd2/journal.c:	new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);

Might be.

Here, I see the following scenario.  With freshly booted server, 1.9Gb RAM,
slabtop shows about 11K entries in buffer_head slab, and about 1.7Gb free RAM.

When starting writing from another machine to this one over nfs, buffer_head
slab grows quite rapidly up to about 450K entries (total size 48940K) and
free memory drops to almost zero -- this happens in first 1..2 minutes
(GigE network, writing from /dev/zero using dd).

The cache does not grow further -- just because there's no free memory for
growing.  On a 4Gb machine it grows up to about 920K objects.

And from time to time during write the same warning occurs.  And slows
down write from ~70Mb/sec (it is almost the actual speed of the target
drive - it can do ~80Mb/sec) to almost zero for several seconds.

> > As far as I can see, the warning itself, while harmless, indicates some
> > deeper problem.  Namely, we shouldn't have an OOM condition - the system
> > is doing nothing but NFS, there's only one NFS client which writes single
> > large file, the system has 2GB (or 4Gb on another machine) RAM.  It should
> > not OOM to start with.
>
> Thanks to the page allocation failure that Justin posted earlier, which 
> shows the state of the available system memory, it shows that the machine 
> truly is oom.  You seem to have isolated that to an enormous amount of 
> buffer_head slab, which is a good start.

It's not really slabs it seems.  In my case the total amount of buffer_heads
is about 49Mb which is very small compared with the amount of memory on the
system.  But as far as I can *guess* buffer_head is just that - head, a
pointer to some other place...  Unwritten or cached data?

Note that the only way to shrink that buffer_head cache back is to remove
the file in question on the server.

> > Well, there ARE side-effects actually.  When the issue happens, the I/O
> > over NFS slows down to almost zero bytes/sec for some while, and resumes
> > slowly after about half a minute - sometimes faster, sometimes slower.
> > Again, the warning itself is harmless, but it shows a deeper issue.  I
> > don't think it's wise to ignore the sympthom -- the actual cause should
> > be fixed instead.  I think.
>
> Since the GFP_ATOMIC allocation cannot trigger reclaim itself, it must 
> rely on other allocations or background writeout to free the memory and 
> this will be considerably slower than a blocking allocation.  The page 
> allocation failure messages from Justin's post indicate there are 0 pages 
> under writeback at the time of oom yet ZONE_NORMAL has reclaimable memory; 
> this is the result of the nonblocking allocation.

So... what's the "consensus" so far?  Just shut up the warning as you
initially proposed?

At least I don't see any immediately alternative.  Well, but I don't know
kernel internals either :)

Thanks!

/mjt
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/