A long overdue fork-bomb defense ?

From: Robert Hailey
Date: Fri Apr 08 2011 - 16:47:18 EST

(Non-subscriber, please CC me in responses)

An idea inspired by the Out-of-Memory Killer, to catch fork bombs at fork-time...

The principal indicator of a fork bomb is that it calls fork() too much, filling up the process table. What makes it nearly impossible to kill manually is that it continuously spawns new processes. It is precisely this odd behavior which should make it exceptionally easy to kill automatically in the kernel.

#1 per-process unsigned integer field "fork_count"
increments at each fork() [if not max_int],
new processes inherit incremented value from parent process, or
zero if parent is init, exec() should *not* clear fork_count

(the above statistic alone might be interesting?)

#2 global unsigned int "fork_alert_level" usually zero
doubles both as a alert state switch & fork_count at which to kill processes
#3 global time value "fork_alert_time"
to return the kernel to standard operation when threat is abated

fork would no longer return ELIMIT (or the like), but follow this pattern:

fork(proc) {
unsigned fork_count=(proc->fork_count+1);
//detect overflow
if (fork_count>proc->fork_count) {
} else {
log("fork_count generation");
if (fork_alert_level) {
if (fork_count >= fork_alert_level) {
signal(KILL, proc) && log('killed ...');
//don't: fork_alert_time=now();
if (now()-fork_alert_time>10 seconds?) {
fork_alert_level=0; //Relax

if (!child) {
//No more processes in process table
Proc highest;
Proc second_place;
for ( p : process_table) {
if (init || protected?) continue;
if (!highest || p->fork_count>highest->fork_count) {
assert(highest->fork_count != second_place->fork_count);
if (afraid_of_killing_innocents) {
fork_alert_level = second_place->fork_count;
} else {
fork_alert_level = second_place->fork_count -
if (fork_count >= fork_alert_level) {
signal(KILL, proc) && log('killed ...');
signal(KILL, highest) && log('killed ...');
} else {
signal(KILL, highest) && log('killed ...');

...continue with fork() logic...

So in normal operation it would only count forks.

Of course, the concern is killing processes that legitimately fork many times over the course of their life (e.g. inetd, cron...). Such a process is not really in danger from it's own children (as they inherit the fork_count), but might be in danger of another process' child.

Seeing that the new systemd puts each service in it's own process group, at first I was thinking this could be easily solved by isolating the fork-based-killing to the process group which contains the most processes at the moment the process table is found to be full.

Does this sound reasonable, or would an out-of-processes-killer also need to scan for session ids or be overly complicated (by uid, by tty)?

Would it even work? thoughts/ideas?

Robert Hailey

Seen: delivering a fork_bomb_defense as a kernel module

Seen: fork bomb defense with OOM-killer by least_common_ancestor

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/