brk() should check randomize_va_space ratherthen CONFIG_COMPAT_BRK

From: Frank Heckenbach
Date: Sun Feb 27 2011 - 07:09:41 EST

The brk() syscall checks CONFIG_COMPAT_BRK to decide whether the brk
area starts just after the end of the code+bss.

Since CONFIG_COMPAT_BRK can be overridden by
/proc/sys/kernel/randomize_va_space, it should rather check this
variable (which is initialized in mm/memory.c from
CONFIG_COMPAT_BRK), see the patch below.

I've tested it with an old libc5 binary: Without the patch, if the
kernel was configured without CONFIG_COMPAT_BRK, the first malloc()
fails with any setting of randomize_va_space. With the patch
applied, the program runs correctly with randomize_va_space < 2, but
not with randomize_va_space == 2, as it should be.

--- linux- 2011-02-25 00:09:00.000000000 +0100
+++ linux- 2011-02-26 01:13:53.000000000 +0100
@@ -252,11 +252,11 @@


- min_brk = mm->end_code;
- min_brk = mm->start_brk;
+ if (randomize_va_space < 2)
+ min_brk = mm->end_code;
+ else
+ min_brk = mm->start_brk;
if (brk < min_brk)
goto out;

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at