Re: Found the commit that causes the OOMs

From: Wu Fengguang
Date: Sun Jun 28 2009 - 10:23:05 EST

Next message: Kay Sievers: "Re: Kernel 2.6.30 and udevd problem"
Previous message: Alberto Gonzalez: "Re: Kernel 2.6.30 and udevd problem"
In reply to: Minchan Kim: "Re: Found the commit that causes the OOMs"
Next in thread: KOSAKI Motohiro: "Re: Found the commit that causes the OOMs"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Sun, Jun 28, 2009 at 09:36:49PM +0800, Minchan Kim wrote:
> On Sun, Jun 28, 2009 at 10:30 PM, Minchan Kim<minchan.kim@xxxxxxxxx> wrote:
> > HI, Wu.
> >
> > On Sun, Jun 28, 2009 at 8:32 PM, Wu Fengguang<fengguang.wu@xxxxxxxxx> wrote:
> >> On Sat, Jun 27, 2009 at 08:54:12PM +0800, Johannes Weiner wrote:
> >>> On Sat, Jun 27, 2009 at 08:12:49AM +0100, David Howells wrote:
> >>> >
> >>> > I've managed to bisect things to find the commit that causes the OOMs. ÂIt's:
> >>> >
> >>> > Â Â commit 69c854817566db82c362797b4a6521d0b00fe1d8
> >>> > Â Â Author: MinChan Kim <minchan.kim@xxxxxxxxx>
> >>> > Â Â Date: Â Tue Jun 16 15:32:44 2009 -0700
> >>> >
> >>> > Â Â Â Â vmscan: prevent shrinking of active anon lru list in case of no swap space V3
> >>> >
> >>> > Â Â Â Â shrink_zone() can deactivate active anon pages even if we don't have a
> >>> > Â Â Â Â swap device. ÂMany embedded products don't have a swap device. ÂSo the
> >>> > Â Â Â Â deactivation of anon pages is unnecessary.
> >>> >
> >>> > Â Â Â Â This patch prevents unnecessary deactivation of anon lru pages. ÂBut, it
> >>> > Â Â Â Â don't prevent aging of anon pages to swap out.
> >>> >
> >>> > Â Â Â Â Signed-off-by: Minchan Kim <minchan.kim@xxxxxxxxx>
> >>> > Â Â Â Â Acked-by: KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx>
> >>> > Â Â Â Â Cc: Johannes Weiner <hannes@xxxxxxxxxxx>
> >>> > Â Â Â Â Acked-by: Rik van Riel <riel@xxxxxxxxxx>
> >>> > Â Â Â Â Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
> >>> > Â Â Â Â Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> >>> >
> >>> > This exhibits the problem. ÂThe previous commit:
> >>> >
> >>> > Â Â commit 35282a2de4e5e4e173ab61aa9d7015886021a821
> >>> > Â Â Author: Brice Goglin <Brice.Goglin@xxxxxxxxxxxx>
> >>> > Â Â Date: Â Tue Jun 16 15:32:43 2009 -0700
> >>> >
> >>> > Â Â Â Â migration: only migrate_prep() once per move_pages()
> >>> >
> >>> > survives 16 iterations of the LTP syscall testsuite without exhibiting the
> >>> > problem.
> >>>
> >>> Here is the patch in question:
> >>>
> >>> diff --git a/mm/vmscan.c b/mm/vmscan.c
> >>> index 7592d8e..879d034 100644
> >>> --- a/mm/vmscan.c
> >>> +++ b/mm/vmscan.c
> >>> @@ -1570,7 +1570,7 @@ static void shrink_zone(int priority, struct zone *zone,
> >>> Â Â Â Â* Even if we did not try to evict anon pages at all, we want to
> >>> Â Â Â Â* rebalance the anon lru active/inactive ratio.
> >>> Â Â Â Â*/
> >>> - Â Â if (inactive_anon_is_low(zone, sc))
> >>> + Â Â if (inactive_anon_is_low(zone, sc) && nr_swap_pages > 0)
> >>> Â Â Â Â Â Â Â shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
> >>>
> >>> Â Â Â throttle_vm_writeout(sc->gfp_mask);
> >>>
> >>> When this was discussed, I think we missed that nr_swap_pages can
> >>> actually get zero on swap systems as well and this should have been
> >>> total_swap_pages - otherwise we also stop balancing the two anon lists
> >>> when swap is _full_ which was not the intention of this change at all.
> >>
> >> Exactly. In Jesse's OOM case, the swap is exhausted.
> >> total_swap_pages is the better choice in this situation.
> >>
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426766] Active_anon:290797 active_file:28 inactive_anon:97034
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426767] Âinactive_file:61 unevictable:11322 dirty:0 writeback:0 unstable:0
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426768] Âfree:3341 slab:13776 mapped:5880 pagetables:6851 bounce:0
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426772] DMA free:7776kB min:40kB low:48kB high:60kB active_anon:556kB inactive_anon:524kB
> >> +active_file:16kB inactive_file:0kB unevictable:0kB present:15340kB pages_scanned:30 all_unreclaimable? no
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426775] lowmem_reserve[]: 0 1935 1935 1935
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426781] DMA32 free:5588kB min:5608kB low:7008kB high:8412kB active_anon:1162632kB
> >> +inactive_anon:387612kB active_file:96kB inactive_file:256kB unevictable:45288kB present:1982128kB pages_scanned:980
> >> +all_unreclaimable? no
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426784] lowmem_reserve[]: 0 0 0 0
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426787] DMA: 64*4kB 77*8kB 45*16kB 18*32kB 4*64kB 2*128kB 2*256kB 3*512kB 1*1024kB
> >> +1*2048kB 0*4096kB = 7800kB
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426796] DMA32: 871*4kB 149*8kB 1*16kB 2*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB
> >> +0*2048kB 0*4096kB = 5588kB
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426804] 151250 total pagecache pages
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426806] 18973 pages in swap cache
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426808] Swap cache stats: add 610640, delete 591667, find 144356/181468
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426810] Free swap Â= 0kB
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.426811] Total swap = 979956kB
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.434828] 507136 pages RAM
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.434831] 23325 pages reserved
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.434832] 190892 pages shared
> >> Jun 18 07:44:53 jbarnes-g45 kernel: [64377.434833] 248816 pages non-shared
> >>
> >>
> >> In David's OOM case, there are two symptoms:
> >> 1) 70000 unaccounted/leaked pages as found by Andrew
> >> Â (plus rather big number of PG_buddy and pagetable pages)
> >> 2) almost zero active_file/inactive_file; small inactive_anon;
> >> Â many slab and active_anon pages.
> >>
> >> In the situation of (2), the slab cache is _under_ scanned. So David
> >> got OOM when vmscan should have squeezed some free pages from the slab
> >> cache. Which is one important side effect of MinChan's patch?
> >
> > My patch's side effect is (2).
> >
> > My guessing is following as.
> >
> > 1. The number of page scanned in shrink_slab is increased in shrink_page_list.
> > And it is doubled for mapped page or swapcache.
> > 2. shrink_page_list is called by shrink_inactive_list
> > 3. shrink_inactive_list is called by shrink_list
> >
> > Look at the shrink_list.
> > If inactive lru list is low, it always call shrink_active_list not
> > shrink_inactive_list in case of anon.
>
> I missed most important point.
> My patch's side effect is that it keeps inactive anon's lru low.
> So I think it is caused by my patch's side effect.

Yes, smaller inactive_anon means smaller (pointless) nr_scanned,
and therefore less slab scans. Strictly speaking, it's not the fault
of your patch. It indicates that the slab scan ratio algorithm should
be updated too :)

We could refine the estimation of "reclaimable" pages like this:

diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 416f748..e9c5b0e 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -167,14 +167,7 @@ static inline unsigned long zone_page_state(struct zone *zone,
}

extern unsigned long global_lru_pages(void);
-
-static inline unsigned long zone_lru_pages(struct zone *zone)
-{
- return (zone_page_state(zone, NR_ACTIVE_ANON)
- + zone_page_state(zone, NR_ACTIVE_FILE)
- + zone_page_state(zone, NR_INACTIVE_ANON)
- + zone_page_state(zone, NR_INACTIVE_FILE));
-}
+extern unsigned long zone_lru_pages(void);

#ifdef CONFIG_NUMA
/*
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 026f452..4281c6f 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2123,10 +2123,31 @@ void wakeup_kswapd(struct zone *zone, int order)

unsigned long global_lru_pages(void)
{
- return global_page_state(NR_ACTIVE_ANON)
- + global_page_state(NR_ACTIVE_FILE)
- + global_page_state(NR_INACTIVE_ANON)
- + global_page_state(NR_INACTIVE_FILE);
+ int nr;
+
+ nr = global_page_state(zone, NR_ACTIVE_FILE) +
+ global_page_state(zone, NR_INACTIVE_FILE);
+
+ if (total_swap_pages)
+ nr += global_page_state(zone, NR_ACTIVE_ANON) +
+ global_page_state(zone, NR_INACTIVE_ANON);
+
+ return nr;
+}
+
+
+unsigned long zone_lru_pages(struct zone *zone)
+{
+ int nr;
+
+ nr = zone_page_state(zone, NR_ACTIVE_FILE) +
+ zone_page_state(zone, NR_INACTIVE_FILE);
+
+ if (total_swap_pages)
+ nr += zone_page_state(zone, NR_ACTIVE_ANON) +
+ zone_page_state(zone, NR_INACTIVE_ANON);
+
+ return nr;
}

#ifdef CONFIG_HIBERNATION
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Kay Sievers: "Re: Kernel 2.6.30 and udevd problem"
Previous message: Alberto Gonzalez: "Re: Kernel 2.6.30 and udevd problem"
In reply to: Minchan Kim: "Re: Found the commit that causes the OOMs"
Next in thread: KOSAKI Motohiro: "Re: Found the commit that causes the OOMs"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]