OOM killer in 2.6.31-rc2

From: Gene Heskett
Date: Mon Jul 06 2009 - 10:56:54 EST

Next message: Alexey Fisher: "Intel BIOS - Corrupted low memory at ffff880000004200"
Previous message: Gregory Haskins: "Re: [KVM PATCH v9 0/5] irqfd fixes and enhancements"
Next in thread: Wu Fengguang: "Re: OOM killer in 2.6.31-rc2"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Greetings all;

I had to hard reset this box just now as there was no response to a ctl-alt-
bksp when X was un-responsive this morning.

I had built a 2nd version of the 2.6.31-rc2 kernel last night when I found the
video stuff appeared to have been moved in the .config and my pcHDTV-3000
cards modules were not being built, but are now. That kernel was installed,
and if at some time in the night a module was needed, it would have been
available, but I can't make a solid connection.

This machine will always be marked as 'tainted' because any bios update that
fixes the very early boot time oops, also leaves me with a machine that will
crash hard in 30 seconds to 3 or 4 hours. The fixes done by the oops make it
generally dead stable for weeks. That oops:

Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-provided physical RAM map:
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 0000000000100000 - 00000000dfee0000 (usable)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000dfee0000 - 00000000dfee3000 (ACPI NVS)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000dfee3000 - 00000000dfef0000 (ACPI data)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000dfef0000 - 00000000dff00000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
Jul 6 10:03:58 coyote kernel: [ 0.000000] DMI 2.4 present.
Jul 6 10:03:58 coyote kernel: [ 0.000000] Phoenix BIOS detected: BIOS may corrupt low RAM, working around it.
Jul 6 10:03:58 coyote kernel: [ 0.000000] last_pfn = 0x120000 max_arch_pfn = 0x1000000
Jul 6 10:03:58 coyote kernel: [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jul 6 10:03:58 coyote kernel: [ 0.000000] ------------[ cut here ]------------
Jul 6 10:03:58 coyote kernel: [ 0.000000] WARNING: at arch/x86/kernel/cpu/mtrr/generic.c:456
generic_get_mtrr+0x12c/0x150()
Jul 6 10:03:58 coyote kernel: [ 0.000000] Hardware name: System Product Name
Jul 6 10:03:58 coyote kernel: [ 0.000000] mtrr: your BIOS has set up an incorrect mask, fixing it up.
Jul 6 10:03:58 coyote kernel: [ 0.000000] Modules linked in:
Jul 6 10:03:58 coyote kernel: [ 0.000000] Pid: 0, comm: swapper Not tainted 2.6.31-rc2 #2
Jul 6 10:03:58 coyote kernel: [ 0.000000] Call Trace:
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c101449c>] ? generic_get_mtrr+0x12c/0x150
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c103693d>] warn_slowpath_common+0x7d/0xe0
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c101449c>] ? generic_get_mtrr+0x12c/0x150
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c1036a13>] warn_slowpath_fmt+0x33/0x50
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c101449c>] generic_get_mtrr+0x12c/0x150
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c1422e1a>] mtrr_trim_uncached_memory+0x85/0x368
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c142191e>] ? mtrr_bp_init+0x1d9/0x2bb
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c141c369>] setup_arch+0x52c/0xa33
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c11c0020>] ? thermal_get_trip_type+0x0/0x9c
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c1418bb4>] start_kernel+0xb2/0x38b
Jul 6 10:03:58 coyote kernel: [ 0.000000] [<c1418394>] i386_start_kernel+0x84/0xb0
Jul 6 10:03:58 coyote kernel: [ 0.000000] ---[ end trace a7919e7f17c0a725 ]---
Jul 6 10:03:58 coyote kernel: [ 0.000000] Scanning 0 areas for low memory corruption
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified physical RAM map:
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000000000000 - 0000000000010000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000000010000 - 000000000009f000 (usable)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 000000000009f000 - 00000000000a0000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000000f0000 - 0000000000100000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000000100000 - 00000000dfee0000 (usable)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000dfee0000 - 00000000dfee3000 (ACPI NVS)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000dfee3000 - 00000000dfef0000 (ACPI data)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000dfef0000 - 00000000dff00000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000f0000000 - 00000000f4000000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000fec00000 - 0000000100000000 (reserved)
Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000100000000 - 0000000120000000 (usable)
Jul 6 10:03:58 coyote kernel: [ 0.000000] init_memory_mapping: 0000000000000000-00000000379fe000
Jul 6 10:03:58 coyote kernel: [ 0.000000] NX (Execute Disable) protection: active

You all have seen this one before, several times. I have asked that
since its a good fix, that the kernel not be marked tainted in that
instance. I would run the asus bios that didn't do that _IF_ it was
stable. 2 newer versions are _not_ stable, this is stable after the fix.

The machine has 4G of ram & is I believe 'pae'

The oom's first stanza:

Jul 6 06:45:01 coyote kernel: [78748.106803] X invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0
Jul 6 06:45:01 coyote kernel: [78748.106808] Pid: 3068, comm: X Tainted: G W 2.6.31-rc2 #1
Jul 6 06:45:01 coyote kernel: [78748.106811] Call Trace:
Jul 6 06:45:01 coyote kernel: [78748.106818] [<c1308513>] ? printk+0x23/0x40
Jul 6 06:45:01 coyote kernel: [78748.106823] [<c107e268>] oom_kill_process+0x178/0x270
Jul 6 06:45:01 coyote kernel: [78748.106827] [<c107e6ad>] ? badness+0x14d/0x220
Jul 6 06:45:01 coyote kernel: [78748.106830] [<c107e8c2>] __out_of_memory+0x142/0x170
Jul 6 06:45:01 coyote kernel: [78748.106834] [<c107e949>] out_of_memory+0x59/0xc0
Jul 6 06:45:01 coyote kernel: [78748.106837] [<c1081d17>] __alloc_pages_nodemask+0x4f7/0x510
Jul 6 06:45:01 coyote kernel: [78748.106841] [<c1081db3>] __get_free_pages+0x23/0x50
Jul 6 06:45:01 coyote kernel: [78748.106845] [<c10bf8b2>] __pollwait+0xb2/0xf0
Jul 6 06:45:01 coyote kernel: [78748.106848] [<c12f6528>] unix_poll+0x28/0xc0
Jul 6 06:45:01 coyote kernel: [78748.106851] [<c1281b7e>] sock_poll+0x1e/0x40
Jul 6 06:45:01 coyote kernel: [78748.106853] [<c10bee8e>] do_select+0x34e/0x6b0
Jul 6 06:45:01 coyote kernel: [78748.106871] [<c10bf800>] ? __pollwait+0x0/0xf0
Jul 6 06:45:01 coyote kernel: [78748.106874] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106877] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106879] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106882] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106884] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106887] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106890] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106892] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106895] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106897] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106900] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106902] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106905] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106907] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106910] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106913] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106915] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106918] [<c10bf8f0>] ? pollwake+0x0/0x90
Jul 6 06:45:01 coyote kernel: [78748.106920] [<c10bf402>] core_sys_select+0x212/0x350
Jul 6 06:45:01 coyote kernel: [78748.106925] [<c10be292>] ? poll_select_set_timeout+0x82/0x90
Jul 6 06:45:01 coyote kernel: [78748.106928] [<c10bf761>] sys_select+0x51/0xf0
Jul 6 06:45:01 coyote kernel: [78748.106931] [<c10031b7>] sysenter_do_call+0x12/0x22
Jul 6 06:45:01 coyote kernel: [78748.106933] Mem-Info:
Jul 6 06:45:01 coyote kernel: [78748.106935] DMA per-cpu:
Jul 6 06:45:01 coyote kernel: [78748.106937] CPU 0: hi: 0, btch: 1 usd: 0
Jul 6 06:45:01 coyote kernel: [78748.106939] CPU 1: hi: 0, btch: 1 usd: 0
Jul 6 06:45:01 coyote kernel: [78748.106941] CPU 2: hi: 0, btch: 1 usd: 0
Jul 6 06:45:01 coyote kernel: [78748.106943] CPU 3: hi: 0, btch: 1 usd: 0
Jul 6 06:45:01 coyote kernel: [78748.106944] Normal per-cpu:
Jul 6 06:45:01 coyote kernel: [78748.106946] CPU 0: hi: 186, btch: 31 usd: 118
Jul 6 06:45:01 coyote kernel: [78748.106948] CPU 1: hi: 186, btch: 31 usd: 171
Jul 6 06:45:01 coyote kernel: [78748.106950] CPU 2: hi: 186, btch: 31 usd: 159
Jul 6 06:45:01 coyote kernel: [78748.106952] CPU 3: hi: 186, btch: 31 usd: 172
Jul 6 06:45:01 coyote kernel: [78748.106954] HighMem per-cpu:
Jul 6 06:45:01 coyote kernel: [78748.106955] CPU 0: hi: 186, btch: 31 usd: 56
Jul 6 06:45:01 coyote kernel: [78748.106957] CPU 1: hi: 186, btch: 31 usd: 20
Jul 6 06:45:01 coyote kernel: [78748.106959] CPU 2: hi: 186, btch: 31 usd: 53
Jul 6 06:45:01 coyote kernel: [78748.106961] CPU 3: hi: 186, btch: 31 usd: 180
Jul 6 06:45:01 coyote kernel: [78748.106965] Active_anon:90702 active_file:136927 inactive_anon:26328
Jul 6 06:45:01 coyote kernel: [78748.106966] inactive_file:1956 unevictable:25 dirty:4 writeback:0 unstable:0
Jul 6 06:45:01 coyote kernel: [78748.106967] free:560899 slab:206505 mapped:19048 pagetables:3220 bounce:0
Jul 6 06:45:01 coyote kernel: [78748.106971] DMA free:3496kB min:64kB low:80kB high:96kB active_anon:0kB
inactive_anon:0kB acti
ve_file:12kB inactive_file:8kB unevictable:0kB present:15804kB pages_scanned:0 all_unreclaimable? yes
Jul 6 06:45:01 coyote kernel: [78748.106983] lowmem_reserve[]: 0 0 25406 25406
Jul 6 06:45:01 coyote kernel: [78748.106988] HighMem free:2236464kB min:512kB low:3928kB high:7348kB
active_anon:362808kB inact
ive_anon:105308kB active_file:547220kB inactive_file:7704kB unevictable:100kB present:3252052kB pages_scanned:0
all_unreclaimabl
e? no
Jul 6 06:45:01 coyote kernel: [78748.106991] lowmem_reserve[]: 0 0 0 0
Jul 6 06:45:01 coyote kernel: [78748.106994] DMA: 310*4kB 204*8kB 27*16kB 6*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048
kB 0*4096kB = 3496kB
Jul 6 06:45:01 coyote kernel: [78748.107002] Normal: 1*4kB 0*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB
1*1024kB 1*2048kB
0*4096kB = 3636kB
Jul 6 06:45:01 coyote kernel: [78748.107009] HighMem: 45888*4kB 44682*8kB 32844*16kB 18479*32kB 6641*64kB 1114*128kB
37*256kB 1
*512kB 1*1024kB 0*2048kB 0*4096kB = 2236464kB
Jul 6 06:45:01 coyote kernel: [78748.107017] 139771 total pagecache pages
Jul 6 06:45:01 coyote kernel: [78748.107019] 616 pages in swap cache
Jul 6 06:45:01 coyote kernel: [78748.107021] Swap cache stats: add 7937, delete 7321, find 2362/2515
Jul 6 06:45:01 coyote kernel: [78748.107023] Free swap = 8360652kB
Jul 6 06:45:01 coyote kernel: [78748.107024] Total swap = 8385912kB
Jul 6 06:45:01 coyote kernel: [78748.121323] 1179632 pages RAM
Jul 6 06:45:01 coyote kernel: [78748.121325] 951810 pages HighMem
Jul 6 06:45:01 coyote kernel: [78748.121327] 146165 pages reserved
Jul 6 06:45:01 coyote kernel: [78748.121328] 149491 pages shared
Jul 6 06:45:01 coyote kernel: [78748.121329] 441615 pages non-shared
Jul 6 06:45:01 coyote kernel: [78748.121332] Out of memory: kill process 2385 (mysqld) score 15451 or a child
Jul 6 06:45:01 coyote kernel: [78748.121334] Killed process 2385 (mysqld)

and continued to:

Jul 6 06:45:01 coyote kernel: [78748.137525] Killed process 30192 (spamd)
Jul 6 06:45:01 coyote kernel: [78748.154292] Killed process 2506 (httpd)
Jul 6 06:45:01 coyote kernel: [78748.170851] Killed process 2507 (httpd)
Jul 6 06:45:01 coyote kernel: [78748.187519] Killed process 2508 (httpd)
Jul 6 06:45:01 coyote kernel: [78748.320880] Killed process 2510 (httpd)
Jul 6 06:45:01 coyote kernel: [78748.337529] Killed process 2511 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.467158] Killed process 948 (spamd)
Jul 6 06:45:05 coyote kernel: [78751.483535] Killed process 963 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.500194] Killed process 964 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.502688] Killed process 965 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.506025] Killed process 2512 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.509376] Killed process 2513 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.512713] Killed process 2514 (httpd)
Jul 6 06:45:05 coyote kernel: [78751.516057] Killed process 1995 (krunner_lock)
Jul 6 06:45:05 coyote kernel: [78751.519481] Killed process 3391 (kwin)
Jul 6 06:45:05 coyote kernel: [78751.522674] Killed process 3396 (plasma)
Jul 6 06:45:05 coyote kernel: [78751.526006] Killed process 6931 (kmail)
Jul 6 06:45:05 coyote kernel: [78751.542786] Killed process 16309 (spamd)
Jul 6 06:45:05 coyote kernel: [78751.546042] Killed process 3421 (krunner)
Jul 6 06:45:05 coyote kernel: [78751.549521] Killed process 3357 (klauncher)
Jul 6 06:45:05 coyote kernel: [78751.552760] Killed process 3526 (kcalc)
Jul 6 06:45:05 coyote kernel: [78751.759570] Killed process 3068 (X)
Jul 6 06:45:05 coyote kernel: [78751.765648] Killed process 3389 (ksmserver)

The oom started at 6:45:01 this morning. mysqld wasn't doing anything & the
only reason its even started is for mythtv, which is how I found my
pcHDTV-3000 was on the missing list even if occupying a slot.

I've no idea if the rebuilt (with v4l drivers now) will also crash.
It feels normal. And looks normal in htop's display, using 538M of 4096M,
no swap used yet.

>From my .config:

# grep MEM .config
CONFIG_SHMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_NOHIGHMEM is not set
# CONFIG_HIGHMEM4G is not set
CONFIG_HIGHMEM64G=y
CONFIG_HIGHMEM=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_DEVKMEM=y
CONFIG_FIX_EARLYCON_MEM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
# CONFIG_MEMSTICK is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_STRICT_DEVMEM=y
CONFIG_HAS_IOMEM=y

In case I have some option miss-set in that, plz advise.

Thanks.

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The NRA is offering FREE Associate memberships to anyone who wants them.
<https://www.nrahq.org/nrabonus/accept-membership.asp>

Mal: "Shouldn't have been you - they shoulda hit us."

Book: "That crossed my mind..."
--"Serenity"

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Alexey Fisher: "Intel BIOS - Corrupted low memory at ffff880000004200"
Previous message: Gregory Haskins: "Re: [KVM PATCH v9 0/5] irqfd fixes and enhancements"
Next in thread: Wu Fengguang: "Re: OOM killer in 2.6.31-rc2"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]