Re: Odd ENOMEM being returned in 3.8-rcX

[Josh Boyer]

On Thu, Feb 07, 2013 at 02:15:02PM -0800, Andrew Morton wrote:
> On Thu, 7 Feb 2013 16:57:42 -0500
> Josh Boyer <jwboyer@xxxxxxxxxx> wrote:
> 
> > Hi All,
> > 
> > We've hit a weird error in Fedora using the 3.8-rcX kernels.  It seems
> > the mock tool is getting back ENOMEM when doing very simple things that
> > normally just work.  The 3.7 kernels on the same userspace work just
> > fine.  It seems just running 'mock init -v' is enough to cause the
> > failure.
> 
> I assume you're not seeing the "page allocation failure" message and
> backtrace.  This means that either

Right.  If I disable our debug options, I see no backtraces at all and
the python app still gets ENOMEM returned.  (See below for those
interested).

> a) it's a __GFP_NOWARN callsite.  This is rare.  Or
> 
> b) it's actually a different error but someone went and overwrote a
>    callee's return value with -ENOMEM.  We do this a lot and it sucks.

We do it in copy_io :\.

> > At first glance it seems copy_io is failing (possibly because
> > get_task_io_context fails), and then the above fallout is printed.  The
> > warning seems fairly valid, but I don't think that is the root of the
> > problem.
> 
> yes, get_task_io_context() might be the place.  Tried adding a few
> error-path printks in there to see what's happening?

Yeah, that's my next step.  I guess I know what I'll be doing tomorrow.

> I can't see anything around there which leaves interrupts disabled
> though.  It's quite likely that there's some code with is forgetting to
> reenable interrupts on a rarely-tested error path, and that ENOMEM is
> tickling the bug.

Right, agreed.  As I said, I think that is mostly a secondary issue.
Hopefully it will be easy to fix once we figure out why we're getting
the ENOMEM error.

Python backtrace below.  Seems to be failing on forking a umount command
after init'ing the chroot.  I can put the full output somewhere if
people are interested.

josh

DEBUG: Executing command: ['/bin/umount', '-n', '-l', '/var/lib/mock/fedora-16-x86_64/root/proc'] with env {'LANG': 'en_US.utf8', 'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'}
Traceback (most recent call last):
  File "/usr/sbin/mock", line 539, in <module>
    def do_buildsrpm(config_opts, chroot, options, args):
  File "/usr/sbin/mock", line 782, in main
    chroot.init()
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.init at 0x00D59320>", line 3, in init
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 279, in init
    self._init()
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._init at 0x00D596E0>", line 3, in _init
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 425, in _init
    self._umountall()
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._umountall at 0x00D5C320>", line 3, in _umountall
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 872, in _umountall
    self.mounts.umountall(nowarn=nowarn)
  File "<peak.util.decorators.rewrap wrapping mockbuild.mounts.umountall at 0x00D43398>", line 3, in umountall
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/mounts.py", line 133, in umountall
    m.umount()
  File "<peak.util.decorators.rewrap wrapping mockbuild.mounts.umount at 0x00D4EC08>", line 3, in umount
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/mounts.py", line 69, in umount
    mockbuild.util.do(cmd)
  File "<peak.util.decorators.rewrap wrapping mockbuild.util.do at 0x00D47140>", line 3, in do
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 323, in do
    preexec_fn = preexec,
  File "/usr/lib64/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1143, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/