Re: OOM behavior in constrained memory situations

[Christoph Lameter]

On Mon, 6 Feb 2006, Andrew Morton wrote:

> The oom-killer is invoked from the page allocator.  A hugetlb pagefault
> won't use the page allocator.  So there shouldn't be an oom-killing on
> hugepage exhaustion.

Right..... and the arch specific fault code (at least ia64) does not call 
the OOM killer.

> I think this comment is just wrong:
> 
> 		/* Logically this is OOM, not a SIGBUS, but an OOM
> 		 * could cause the kernel to go killing other
> 		 * processes which won't help the hugepage situation
> 		 * at all (?) */
> 
> A VM_FAULT_OOM from there won't cause the oom-killer to do anything.  We
> should return VM_FAULT_OOM and let do_page_fault() commit suicide with
> SIGKILL.

Drop my patch that adds the comments explaining the bus error and add this 
fix instead. This will terminate an application with out of memory instead 
of bus error and remove the comment that you mentioned.

Signed-off-by: Christoph Lameter <clameter@xxxxxxx>

Index: linux-2.6.16-rc2/mm/hugetlb.c
===================================================================
--- linux-2.6.16-rc2.orig/mm/hugetlb.c	2006-02-02 22:03:08.000000000 -0800
+++ linux-2.6.16-rc2/mm/hugetlb.c	2006-02-06 16:02:53.000000000 -0800
@@ -391,12 +391,7 @@ static int hugetlb_cow(struct mm_struct 
 
 	if (!new_page) {
 		page_cache_release(old_page);
-
-		/* Logically this is OOM, not a SIGBUS, but an OOM
-		 * could cause the kernel to go killing other
-		 * processes which won't help the hugepage situation
-		 * at all (?) */
-		return VM_FAULT_SIGBUS;
+		return VM_FAULT_OOM;
 	}
 
 	spin_unlock(&mm->page_table_lock);
@@ -444,6 +439,7 @@ retry:
 		page = alloc_huge_page(vma, address);
 		if (!page) {
 			hugetlb_put_quota(mapping);
+			ret = VM_FAULT_OOM;
 			goto out;
 		}
 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/