[PATCH] mm: nommu: add sysctl_max_map_count() check for do_mmap()

[Hajime Tazaki]

The sysctl variable vm.max_map_count (sysctl_max_map_count) is not
expose under !MMU case but used it wit the default vaule
DEFAULT_MAX_MAP_COUNT as a limit of count.  This is currently used when
a vma entry is split into two chunks (split_vma()) but not used when
allocated (do_mmap()).  As a result, even if users request a large
number of allocate memory, it will keep allocating until OOM happens.

This commit introduces a check at the begining of do_mmap in nommu.c to
prevent this situation.

This is detected with a LTP (Linux Test Project) test, which linked
below.

Link:
https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/munmap/munmap04.c
Signed-off-by: Hajime Tazaki <thehajime@xxxxxxxxx>
---
 mm/nommu.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/nommu.c b/mm/nommu.c
index 5d461ddc3405..815ebefddfae 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1035,6 +1035,9 @@ unsigned long do_mmap(struct file *file,
 	if (ret < 0)
 		return ret;
 
+	if (current->mm->map_count >= get_sysctl_max_map_count())
+		return -ENOMEM;
+
 	/* we ignore the address hint */
 	addr = 0;
 	len = PAGE_ALIGN(len);
-- 
2.43.0