Re: Linux 3.19-rc3
From: Mark Rutland
Date: Fri Jan 09 2015 - 12:57:59 EST
On Fri, Jan 09, 2015 at 02:27:06PM +0000, Mark Langsdorf wrote:
> On 01/09/2015 08:19 AM, Steve Capper wrote:
> > On 9 January 2015 at 12:13, Mark Rutland <mark.rutland@xxxxxxx> wrote:
> >> On Thu, Jan 08, 2015 at 12:51:31PM +0000, Mark Langsdorf wrote:
> >>> I'm consistently getting an out of memory killer triggered when
> >>> compiling the kernel (make -j 16 -s) on a 16 core ARM64 system
> >>> with 16 GB of memory. This doesn't happen when running a 3.18
> >>> kernel.
> >>>
> >>> I'm going to start bisecting the failure now, but here's the crash
> >>> log in case someone can see something obvious in it.
> >>
> >> FWIW I've just reproduced this with v3.19-rc3 defconfig +
> >> CONFIG_ARM64_64K_PAGES=y by attempting a git clone of mainline. My
> >> system has 16GB of RAM and 6 CPUs.
> >>
> >> I have a similarly dodgy looking number of pages reserved
> >> (18446744073709544451 A.K.A. -7165). Log below.
> >>
> >
> > I think the negative page reserved count is a consequence of another bug.
> >
> > We have the following reporting code in lib/show_mem.c:
> > #ifdef CONFIG_CMA
> > printk("%lu pages reserved\n", (reserved - totalcma_pages));
> > printk("%lu pages cma reserved\n", totalcma_pages);
> > #else
> >
> > With totalcma_pages being reported as 8192, that would account for the
> > -7000ish values reported.
> >
> > That change appears to have come from:
> > 49abd8c lib/show_mem.c: add cma reserved information
> >
> > Is the quickest way to exacerbate this OOM a kernel compile?
>
> I haven't really tried to characterize this. Compiling a kernel
> on a 64K page machine causes a failure reasonably quickly and
> doesn't require a lot of thought. I think that time spent finding
> a faster reproducer wouldn't pay off.
I wasn't able to trigger the issue again with git, and the only way I've
managed to trigger the issue is repeatedly building the kernel in a
loop:
while true; do
git clean -fdx > /dev/null 2>&1;
make defconfig > /dev/null 2>&1;
make > /dev/null > 2>&1;
done
Which after a while died:
-bash: fork: Cannot allocate memory
I didn't see anything interesting in dmesg, but I was able to get at
/proc/meminfo:
MemTotal: 16695168 kB
MemFree: 998336 kB
MemAvailable: 325568 kB
Buffers: 51200 kB
Cached: 236224 kB
SwapCached: 0 kB
Active: 14970880 kB
Inactive: 580288 kB
Active(anon): 14834496 kB
Inactive(anon): 5760 kB
Active(file): 136384 kB
Inactive(file): 574528 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 448 kB
Writeback: 0 kB
AnonPages: 22400 kB
Mapped: 10240 kB
Shmem: 8768 kB
Slab: 63744 kB
SReclaimable: 27072 kB
SUnreclaim: 36672 kB
KernelStack: 1824 kB
PageTables: 3776 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8347584 kB
Committed_AS: 50368 kB
VmallocTotal: 2142764992 kB
VmallocUsed: 283264 kB
VmallocChunk: 2142387200 kB
AnonHugePages: 0 kB
CmaTotal: 524288 kB
CmaFree: 128 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
And also magic-sysrq m:
SysRq : Show Memory
Mem-Info:
DMA per-cpu:
CPU 0: hi: 6, btch: 1 usd: 1
CPU 1: hi: 6, btch: 1 usd: 1
CPU 2: hi: 6, btch: 1 usd: 1
CPU 3: hi: 6, btch: 1 usd: 3
CPU 4: hi: 6, btch: 1 usd: 5
CPU 5: hi: 6, btch: 1 usd: 5
Normal per-cpu:
CPU 0: hi: 6, btch: 1 usd: 0
CPU 1: hi: 6, btch: 1 usd: 5
CPU 2: hi: 6, btch: 1 usd: 1
CPU 3: hi: 6, btch: 1 usd: 5
CPU 4: hi: 6, btch: 1 usd: 5
CPU 5: hi: 6, btch: 1 usd: 5
active_anon:231780 inactive_anon:90 isolated_anon:0
active_file:2131 inactive_file:8977 isolated_file:0
unevictable:0 dirty:8 writeback:0 unstable:0
free:15601 slab_reclaimable:423 slab_unreclaimable:573
mapped:160 shmem:137 pagetables:59 bounce:0
free_cma:2
DMA free:302336kB min:208000kB low:259968kB high:312000kB active_anon:3618432kB inactive_anon:768kB active_file:34432kB inactive_file:131584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4166528kB mlocked:0kB dirty:192kB writeback:0kB mapped:4736kB shmem:1024kB slab_reclaimable:5184kB slab_unreclaimable:3328kB kernel_stack:0kB pagetables:1600kB unstable:0kB bounce:0kB free_cma:128kB writeback_tmp:0kB pages_scanned:1208448 all_unreclaimable? yes
lowmem_reserve[]: 0 764 764
Normal free:696128kB min:625472kB low:781824kB high:938176kB active_anon:11215488kB inactive_anon:4992kB active_file:101952kB inactive_file:442944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12582912kB managed:12528640kB mlocked:0kB dirty:320kB writeback:0kB mapped:5504kB shmem:7744kB slab_reclaimable:21888kB slab_unreclaimable:33344kB kernel_stack:1840kB pagetables:2176kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3331648 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
DMA: 42*64kB (MRC) 37*128kB (R) 6*256kB (R) 5*512kB (R) 2*1024kB (R) 3*2048kB (R) 1*4096kB (R) 0*8192kB 1*16384kB (R) 0*32768kB 0*65536kB 0*131072kB 1*262144kB (R) 0*524288kB = 302336kB
Normal: 280*64kB (MR) 40*128kB (R) 5*256kB (R) 4*512kB (R) 6*1024kB (R) 4*2048kB (R) 1*4096kB (R) 1*8192kB (R) 1*16384kB (R) 1*32768kB (R) 1*65536kB (R) 0*131072kB 0*262144kB 1*524288kB (R) = 691968kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
4492 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
261888 pages RAM
0 pages HighMem/MovableOnly
18446744073709544450 pages reserved
8192 pages cma reserved
I also ran ps aux, but I didn't see any stale tasks lying around, nor
did any remaining tasks seem to account for all that active anonymous
memory.
I'll see if I can reproduce on x86.
Thanks,
Mark.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/