[tip:x86/mm] x86/mmap, ASLR: Do not treat unlimited-stack tasks as legacy mmap

[tip-bot for Michal Hocko]

Commit-ID:  4a06370bcb674af88679a4f2c5c87c3e40688935
Gitweb:     http://git.kernel.org/tip/4a06370bcb674af88679a4f2c5c87c3e40688935
Author:     Michal Hocko <mhocko@xxxxxxxx>
AuthorDate: Wed, 14 Jun 2017 10:22:18 +0200
Committer:  Ingo Molnar <mingo@xxxxxxxxxx>
CommitDate: Sat, 24 Jun 2017 08:39:16 +0200

x86/mmap, ASLR: Do not treat unlimited-stack tasks as legacy mmap

Since the following commit in 2008:

  cc503c1b43e0 ("x86: PIE executable randomization")

We added a heuristics to treat applications with RLIMIT_STACK configured
to unlimited as legacy. This means:

 a) set the mmap_base to 1/3 of address space + randomization and
 b) mmap from bottom to top.

This makes some sense as it allows the stack to grow really large. On the
other hand it reduces the address space usable for default mmaps
(without address hint) quite a lot.

We have received a bug report that SAP HANA workload has hit into this
limitation.

We could argue that the user just got what he asked for when setting
up the unlimited stack but to be realistic growing stack up to 1/6
TASK_SIZE (allowed by mmap_base) is pretty much unimited in the real
life. This would give mmap 20TB of additional address space which is
quite nice. Especially when it is much more likely to use that address
space than the reserved stack.

Digging into the history the original implementation of the randomization:

  8817210d4d96 ("[PATCH] x86_64: Flexmap for 32bit and randomized mappings for 64bit")

didn't have this restriction.

So let's try and remove this assumption - hopefully nothing breaks.

Signed-off-by: Michal Hocko <mhocko@xxxxxxxx>
Acked-by: Jiri Kosina <jkosina@xxxxxxx>
Acked-by: Oleg Nesterov <oleg@xxxxxxxxxx>
Cc: Dave Jones <davej@xxxxxxxxxxxxxxxxx>
Cc: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Cc: akpm@xxxxxxxxxxxxxxxxxxxx
Cc: hughd@xxxxxxxxxx
Cc: linux-mm@xxxxxxxxx
Cc: will.deacon@xxxxxxx
Link: http://lkml.kernel.org/r/tip-86b110d2ae6365ce91cabd37588bc8611770421a@xxxxxxxxxxxxxx
[ So I've applied this to tip:x86/mm with a wider Cc: list - if anyone objects to this change please holler. ]
Signed-off-by: Ingo Molnar <mingo@xxxxxxxxxx>
---
 arch/x86/mm/mmap.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
index 19ad095..797295e 100644
--- a/arch/x86/mm/mmap.c
+++ b/arch/x86/mm/mmap.c
@@ -74,9 +74,6 @@ static int mmap_is_legacy(void)
 	if (current->personality & ADDR_COMPAT_LAYOUT)
 		return 1;
 
-	if (rlimit(RLIMIT_STACK) == RLIM_INFINITY)
-		return 1;
-
 	return sysctl_legacy_va_layout;
 }