Re: [PATCH 0/2] exec: allow core_pipe recursion check to look for avalue of 1 rather than 0 (v2)

From: Neil Horman
Date: Fri Jan 29 2010 - 10:10:42 EST

Ok, version two of this patch. I've cleaned up the comments, checkpatched,
rediffed to the latest -mm, etc. More interestingly, I've taken Olegs changes
into account in this version of the patch. By modifying Andi's work slightly,
I've introduced a new init fptr to the usermodehelper api, which lets a caller
customize the process that will be doing the helping. In the case of the
do_coredump path its now used to create the read/write pipes. This allows us to
remove the stdin specifics from the usermodehelper internals and factor our
call_usermodehelper_pipe entirely. I've tested this myself using abrt on
Fedora, with good results

So, about 6 months ago, I made a set of changes to how the
core-dump-to-a-pipe feature in the kernel works. We had reports of several
races, including some reports of apps bypassing our recursion check so that a
process that was forked as part of a core_pattern setup could infinitely crash
and refork until the system crashed.

We fixes those by improving our recursion checks. The new check
basically refuses to fork a process if its core limit is zero, which works well.

Unfortunately, I've been getting grief from maintainer of user space
programs that are inserted as the forked process of core_pattern. They contend
that in order for their programs (such as abrt and apport) to work, all the
running processes in a system must have their core limits set to a non-zero
value, to which I say 'yes'. I did this by design, and think thats the right
way to do things.

But I've been asked to ease this burden on user space enough times that
I thought I would take a look at it. The first suggestion was to make the
recursion check fail on a non-zero 'special' number, like one. That way the
core collector process could set its core size ulimit to 1, and enable the
kernel's recursion detection. This isn't a bad idea on the surface, but I don't
like it since its opt-in, in that if a program like abrt or apport has a bug and
fails to set such a core limit, we're left with a recursively crashing system

So I've come up with this. What I've done is modify the
call_usermodehelper api such that an extra parameter is added, a function
pointer which will be called by the user helper task, after it forks, but before
it exec's the required process. This will give the caller the opportunity to
get a call back in the processes context, allowing it to do whatever it needs to
to the process in the kernel prior to exec-ing the user space code. In the case
of do_coredump, this callback is ues to set the core ulimit of the helper
process to 1. This elimnates the opt-in problem that I had above, as it allows
the ulimit for core sizes to be set to the value of 1, which is what the
recursion check looks for in do_coredump.

This patch has been tested successfully by some of the Abrt maintainers
in fedora, with good results. Patch applies to the latest -mm tree as of this


Signed-off-by: Neil Horman <nhorman@xxxxxxxxxxxxx>
Tested-by: Neil Horman <nhorman@xxxxxxxxxx>
CC: Ingo Molnar <mingo@xxxxxxxxxx>
CC: drbd-dev@xxxxxxxxxxxxxxxx
CC: Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx>
CC: Thomas Sailer <t.sailer@xxxxxxxxxxxxxx>
CC: Adam Belay <abelay@xxxxxxx>
CC: Greg Kroah-Hartman <gregkh@xxxxxxx>
CC: Michal Januszewski <spock@xxxxxxxxxx>
CC: Al Viro <viro@xxxxxxxxxxxxxxxxxx>
CC: Neil Brown <neilb@xxxxxxx>
CC: Mark Fasheh <mfasheh@xxxxxxxx>
CC: Paul Menage <menage@xxxxxxxxxx>
CC: Stephen Hemminger <shemminger@xxxxxxxxxxxxxxxxxxxx>
CC: Kentaro Takeda <takedakn@xxxxxxxxxxxxx>
CC: Oleg Nesterov <oleg@xxxxxxxxxx
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at