Kernel 2.6.30: Memory/XFS leak, OOM killer kills many processes

From: Justin Piszcz
Date: Thu Jun 11 2009 - 12:15:27 EST

Hello,

I have a daily cron that backs up my root filesystem using xfsdump, it has remain unchanged for at least 7-10 kernel versions. When I migrated to 2.6.30, when the xfsdump ran at its scheduled time, nearly all of my processes were killed due to an OOM situation, I can reproduce the situation.

Kernel: 2.6.30
Dist: Debian Testing
xfsdump: 2.2.48-1

Full dmesg:
http://home.comcast.net/~jpiszcz/20090611/dmesg.txt

Regular boot messages:
[ 12.641483] DVB: registering adapter 0 frontend 0 (Samsung S5H1411 QAM/8VSB Frontend)...
[ 12.647108] tda829x 1-0042: type set to tda8295
[ 12.685552] tda18271 1-0060: attaching existing instance

Then I ran xfsdump:
xfsdump -e -L title -M title -f /nfs/path/root.img

Soon after:
[ 94.950729] xfsdump invoked oom-killer: gfp_mask=0x2d0, order=3, oomkilladj=0
[ 94.954690] Pid: 2736, comm: xfsdump Not tainted 2.6.30 #1
[ 94.958662] Call Trace:
[ 94.962581] [<c0178a6c>] ? oom_kill_process+0xac/0x240
[ 94.966544] [<c017900d>] ? badness+0x17d/0x2b0
[ 94.970454] [<c0179284>] ? __out_of_memory+0x144/0x170
[ 94.974351] [<c0179304>] ? out_of_memory+0x54/0xb0
[ 94.978196] [<c017c142>] ? __alloc_pages_internal+0x442/0x480
[ 94.982076] [<c019b032>] ? cache_alloc_refill+0x2b2/0x500
[ 94.985966] [<c019b349>] ? __kmalloc+0xc9/0xd0
[ 94.989846] [<c02d6e39>] ? kmem_alloc+0x59/0xf0
[ 94.993727] [<c02d6f51>] ? kmem_zalloc+0x11/0x40
[ 94.997575] [<c029b56a>] ? xfs_getbmap+0x19a/0x7a0
[ 95.001398] [<c02928ab>] ? xfs_bmap_search_multi_extents+0x7b/0x100
[ 95.005251] [<c0292997>] ? xfs_bmap_search_extents+0x67/0x120
[ 95.009075] [<c04b3789>] ? io_schedule+0x29/0x40
[ 95.012884] [<c02dbfd0>] ? xfs_getbmapx_format+0x0/0x30
[ 95.016690] [<c04b3b0b>] ? __wait_on_bit_lock+0x5b/0x80
[ 95.020502] [<c01a01bc>] ? file_move+0x1c/0x60
[ 95.024342] [<c019d651>] ? __dentry_open+0xf1/0x2c0
[ 95.028200] [<c02dca7e>] ? xfs_ioc_getbmapx+0x6e/0x90
[ 95.032076] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.035954] [<c02ddce2>] ? xfs_file_ioctl+0x6c2/0x7d0
[ 95.039835] [<c02e0111>] ? xfs_read+0x171/0x290
[ 95.043707] [<c02b50f0>] ? xfs_iunlock+0x70/0xb0
[ 95.047550] [<c02d4192>] ? xfs_free_eofblocks+0x122/0x310
[ 95.051383] [<c02dd620>] ? xfs_file_ioctl+0x0/0x7d0
[ 95.055209] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.059049] [<c01abf8b>] ? vfs_ioctl+0x2b/0x90
[ 95.062866] [<c01ac06b>] ? do_vfs_ioctl+0x7b/0x590
[ 95.066668] [<c02f6f67>] ? _atomic_dec_and_lock+0x47/0x70
[ 95.070487] [<c01af415>] ? dput+0xd5/0x120
[ 95.074231] [<c01b54a8>] ? mntput_no_expire+0x18/0x110
[ 95.077912] [<c01ac5bd>] ? sys_ioctl+0x3d/0x70
[ 95.081589] [<c01220a1>] ? syscall_call+0x7/0xb
[ 95.085252] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.088922] [<c0120000>] ? vmx_create_vcpu+0x100/0x800
[ 95.092534] Mem-Info:
[ 95.096071] DMA per-cpu:
[ 95.099563] CPU 0: hi: 0, btch: 1 usd: 0
[ 95.103056] CPU 1: hi: 0, btch: 1 usd: 0
[ 95.106458] Normal per-cpu:
[ 95.109747] CPU 0: hi: 186, btch: 31 usd: 0
[ 95.113018] CPU 1: hi: 186, btch: 31 usd: 24
[ 95.116248] HighMem per-cpu:
[ 95.119387] CPU 0: hi: 186, btch: 31 usd: 0
[ 95.122488] CPU 1: hi: 186, btch: 31 usd: 29
[ 95.125561] Active_anon:4735 active_file:1703 inactive_anon:972
[ 95.125562] inactive_file:376179 unevictable:0 dirty:370 writeback:0 unstable:272
[ 95.125563] free:445875 slab:202789 mapped:3133 pagetables:302 bounce:0
[ 95.135127] DMA free:3764kB min:284kB low:352kB high:424kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:0 all_unreclaimable? no
[ 95.142051] lowmem_reserve[]: 0 867 4038 4038
[ 95.145533] Normal free:35440kB min:16096kB low:20120kB high:24144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB present:887976kB pages_scanned:0 all_unreclaimable? no
[ 95.152926] lowmem_reserve[]: 0 0 25374 25374
[ 95.156652] HighMem free:1744296kB min:512kB low:15228kB high:29948kB active_anon:18940kB inactive_anon:3888kB active_file:6812kB inactive_file:1504704kB unevictable:0kB present:3247956kB pages_scanned:0 all_unreclaimable? no
[ 95.164621] lowmem_reserve[]: 0 0 0 0
[ 95.168632] DMA: 7*4kB 5*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3764kB
[ 95.172797] Normal: 1948*4kB 1448*8kB 861*16kB 12*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 35456kB
[ 95.177043] HighMem: 349*4kB 3582*8kB 5821*16kB 7689*32kB 5889*64kB 3020*128kB 113*256kB 2*512kB 2*1024kB 3*2048kB 140*4096kB = 1744276kB
[ 95.181388] 377922 total pagecache pages
[ 95.185665] 0 pages in swap cache
[ 95.189858] Swap cache stats: add 0, delete 0, find 0/0
[ 95.194006] Free swap = 2056280kB
[ 95.198125] Total swap = 2056280kB
[ 95.211202] 1310720 pages RAM
[ 95.215282] 1082882 pages HighMem
[ 95.219354] 274370 pages reserved
[ 95.223398] 13360 pages shared
[ 95.227421] 412265 pages non-shared
[ 95.231430] Out of memory: kill process 2623 (sshd) score 2840 or a child
[ 95.235525] Killed process 2624 (bash)
[ 95.246873] xfsdump invoked oom-killer: gfp_mask=0x2d0, order=3, oomkilladj=0
[ 95.251069] Pid: 2736, comm: xfsdump Not tainted 2.6.30 #1
[ 95.255273] Call Trace:
[ 95.259430] [<c0178a6c>] ? oom_kill_process+0xac/0x240
[ 95.263650] [<c017900d>] ? badness+0x17d/0x2b0
[ 95.267849] [<c0179284>] ? __out_of_memory+0x144/0x170
[ 95.272052] [<c0179304>] ? out_of_memory+0x54/0xb0
[ 95.276249] [<c017c142>] ? __alloc_pages_internal+0x442/0x480
[ 95.280467] [<c019b032>] ? cache_alloc_refill+0x2b2/0x500
[ 95.284699] [<c019b349>] ? __kmalloc+0xc9/0xd0
[ 95.288900] [<c02d6e39>] ? kmem_alloc+0x59/0xf0
[ 95.293096] [<c02d6f51>] ? kmem_zalloc+0x11/0x40
[ 95.297278] [<c029b56a>] ? xfs_getbmap+0x19a/0x7a0
[ 95.301462] [<c02928ab>] ? xfs_bmap_search_multi_extents+0x7b/0x100
[ 95.305690] [<c0292997>] ? xfs_bmap_search_extents+0x67/0x120
[ 95.309907] [<c04b3789>] ? io_schedule+0x29/0x40
[ 95.314124] [<c02dbfd0>] ? xfs_getbmapx_format+0x0/0x30
[ 95.318354] [<c04b3b0b>] ? __wait_on_bit_lock+0x5b/0x80
[ 95.322580] [<c01a01bc>] ? file_move+0x1c/0x60
[ 95.326805] [<c019d651>] ? __dentry_open+0xf1/0x2c0
[ 95.331029] [<c02dca7e>] ? xfs_ioc_getbmapx+0x6e/0x90
[ 95.335252] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.339471] [<c02ddce2>] ? xfs_file_ioctl+0x6c2/0x7d0
[ 95.343688] [<c02e0111>] ? xfs_read+0x171/0x290
[ 95.347902] [<c02b50f0>] ? xfs_iunlock+0x70/0xb0
[ 95.352106] [<c02d4192>] ? xfs_free_eofblocks+0x122/0x310
[ 95.356329] [<c02dd620>] ? xfs_file_ioctl+0x0/0x7d0
[ 95.360523] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.364744] [<c01abf8b>] ? vfs_ioctl+0x2b/0x90
[ 95.368964] [<c01ac06b>] ? do_vfs_ioctl+0x7b/0x590
[ 95.373215] [<c02f6f67>] ? _atomic_dec_and_lock+0x47/0x70
[ 95.377486] [<c01af415>] ? dput+0xd5/0x120
[ 95.381746] [<c01b54a8>] ? mntput_no_expire+0x18/0x110
[ 95.386050] [<c01ac5bd>] ? sys_ioctl+0x3d/0x70
[ 95.390346] [<c01220a1>] ? syscall_call+0x7/0xb
[ 95.394632] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.398961] [<c0120000>] ? vmx_create_vcpu+0x100/0x800
[ 95.403256] Mem-Info:
[ 95.407483] DMA per-cpu:
[ 95.411649] CPU 0: hi: 0, btch: 1 usd: 0
[ 95.415715] CPU 1: hi: 0, btch: 1 usd: 0
[ 95.419606] Normal per-cpu:
[ 95.423454] CPU 0: hi: 186, btch: 31 usd: 11
[ 95.427208] CPU 1: hi: 186, btch: 31 usd: 13
[ 95.430786] HighMem per-cpu:
[ 95.434324] CPU 0: hi: 186, btch: 31 usd: 158
[ 95.437762] CPU 1: hi: 186, btch: 31 usd: 0
[ 95.441028] Active_anon:3884 active_file:1703 inactive_anon:972
[ 95.441029] inactive_file:376429 unevictable:0 dirty:518 writeback:0 unstable:272
[ 95.441030] free:446299 slab:202878 mapped:3133 pagetables:302 bounce:0
[ 95.450772] DMA free:3764kB min:284kB low:352kB high:424kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:0 all_unreclaimable? no
[ 95.457350] lowmem_reserve[]: 0 867 4038 4038
[ 95.460630] Normal free:35236kB min:16096kB low:20120kB high:24144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:887976kB pages_scanned:0 all_unreclaimable? no
[ 95.467655] lowmem_reserve[]: 0 0 25374 25374
[ 95.471228] HighMem free:1746196kB min:512kB low:15228kB high:29948kB active_anon:15536kB inactive_anon:3888kB active_file:6812kB inactive_file:1505740kB unevictable:0kB present:3247956kB pages_scanned:0 all_unreclaimable? no
[ 95.478934] lowmem_reserve[]: 0 0 0 0
[ 95.482849] DMA: 7*4kB 5*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3764kB
[ 95.486958] Normal: 1957*4kB 1449*8kB 863*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 35212kB
[ 95.491174] HighMem: 409*4kB 3632*8kB 5839*16kB 7713*32kB 5891*64kB 3021*128kB 113*256kB 2*512kB 2*1024kB 3*2048kB 140*4096kB = 1746228kB
[ 95.495476] 378156 total pagecache pages
[ 95.499695] 0 pages in swap cache
[ 95.503863] Swap cache stats: add 0, delete 0, find 0/0
[ 95.508025] Free swap = 2056280kB
[ 95.512156] Total swap = 2056280kB
[ 95.525018] 1310720 pages RAM
[ 95.529107] 1082882 pages HighMem
[ 95.533174] 274370 pages reserved
[ 95.537230] 12393 pages shared
[ 95.541262] 411399 pages non-shared
[ 95.545282] Out of memory: kill process 2506 (sshd) score 2839 or a child
[ 95.549384] Killed process 2508 (bash)
[ 95.554123] xfsdump invoked oom-killer: gfp_mask=0x2d0, order=3, oomkilladj=0
[ 95.558391] Pid: 2736, comm: xfsdump Not tainted 2.6.30 #1
[ 95.562595] Call Trace:
[ 95.566756] [<c0178a6c>] ? oom_kill_process+0xac/0x240
[ 95.570971] [<c017900d>] ? badness+0x17d/0x2b0
[ 95.575168] [<c0179284>] ? __out_of_memory+0x144/0x170
[ 95.579370] [<c0179304>] ? out_of_memory+0x54/0xb0
[ 95.583564] [<c017c142>] ? __alloc_pages_internal+0x442/0x480
[ 95.587778] [<c019b032>] ? cache_alloc_refill+0x2b2/0x500
[ 95.591997] [<c019b349>] ? __kmalloc+0xc9/0xd0
[ 95.596205] [<c02d6e39>] ? kmem_alloc+0x59/0xf0
[ 95.600408] [<c02d6f51>] ? kmem_zalloc+0x11/0x40
[ 95.604600] [<c029b56a>] ? xfs_getbmap+0x19a/0x7a0
[ 95.608790] [<c02928ab>] ? xfs_bmap_search_multi_extents+0x7b/0x100
[ 95.613022] [<c0292997>] ? xfs_bmap_search_extents+0x67/0x120
[ 95.617243] [<c04b3789>] ? io_schedule+0x29/0x40
[ 95.621467] [<c02dbfd0>] ? xfs_getbmapx_format+0x0/0x30
[ 95.625702] [<c04b3b0b>] ? __wait_on_bit_lock+0x5b/0x80
[ 95.629936] [<c01a01bc>] ? file_move+0x1c/0x60
[ 95.634169] [<c019d651>] ? __dentry_open+0xf1/0x2c0
[ 95.638400] [<c02dca7e>] ? xfs_ioc_getbmapx+0x6e/0x90
[ 95.642628] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.646853] [<c02ddce2>] ? xfs_file_ioctl+0x6c2/0x7d0
[ 95.651078] [<c02e0111>] ? xfs_read+0x171/0x290
[ 95.655300] [<c02b50f0>] ? xfs_iunlock+0x70/0xb0
[ 95.659511] [<c02d4192>] ? xfs_free_eofblocks+0x122/0x310
[ 95.663741] [<c02dd620>] ? xfs_file_ioctl+0x0/0x7d0
[ 95.667939] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.672165] [<c01abf8b>] ? vfs_ioctl+0x2b/0x90
[ 95.676390] [<c01ac06b>] ? do_vfs_ioctl+0x7b/0x590
[ 95.680651] [<c02f6f67>] ? _atomic_dec_and_lock+0x47/0x70
[ 95.684929] [<c01af415>] ? dput+0xd5/0x120
[ 95.689194] [<c01b54a8>] ? mntput_no_expire+0x18/0x110
[ 95.693505] [<c01ac5bd>] ? sys_ioctl+0x3d/0x70
[ 95.697806] [<c01220a1>] ? syscall_call+0x7/0xb
[ 95.702093] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.706426] [<c0120000>] ? vmx_create_vcpu+0x100/0x800
[ 95.710722] Mem-Info:
[ 95.714947] DMA per-cpu:
[ 95.719110] CPU 0: hi: 0, btch: 1 usd: 0
[ 95.723172] CPU 1: hi: 0, btch: 1 usd: 0
[ 95.727058] Normal per-cpu:
[ 95.730902] CPU 0: hi: 186, btch: 31 usd: 0
[ 95.734648] CPU 1: hi: 186, btch: 31 usd: 7
[ 95.738221] HighMem per-cpu:
[ 95.741750] CPU 0: hi: 186, btch: 31 usd: 0
[ 95.745178] CPU 1: hi: 186, btch: 31 usd: 109
[ 95.748434] Active_anon:3070 active_file:1777 inactive_anon:972
[ 95.748435] inactive_file:376392 unevictable:0 dirty:518 writeback:0 unstable:272
[ 95.748436] free:447166 slab:202878 mapped:3133 pagetables:277 bounce:0
[ 95.758139] DMA free:3764kB min:284kB low:352kB high:424kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:0 all_unreclaimable? no
[ 95.764686] lowmem_reserve[]: 0 867 4038 4038
[ 95.767953] Normal free:35304kB min:16096kB low:20120kB high:24144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:887976kB pages_scanned:0 all_unreclaimable? no
[ 95.774949] lowmem_reserve[]: 0 0 25374 25374
[ 95.778505] HighMem free:1749596kB min:512kB low:15228kB high:29948kB active_anon:12280kB inactive_anon:3888kB active_file:7108kB inactive_file:1505592kB unevictable:0kB present:3247956kB pages_scanned:0 all_unreclaimable? no
[ 95.786177] lowmem_reserve[]: 0 0 0 0
[ 95.790072] DMA: 7*4kB 5*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3764kB
[ 95.794159] Normal: 1961*4kB 1445*8kB 866*16kB 6*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 35372kB
[ 95.798354] HighMem: 740*4kB 3815*8kB 5877*16kB 7713*32kB 5891*64kB 3021*128kB 113*256kB 2*512kB 2*1024kB 3*2048kB 140*4096kB = 1749624kB
[ 95.802636] 378181 total pagecache pages
[ 95.806838] 0 pages in swap cache
[ 95.810988] Swap cache stats: add 0, delete 0, find 0/0
[ 95.815132] Free swap = 2056280kB
[ 95.819246] Total swap = 2056280kB
[ 95.831982] 1310720 pages RAM
[ 95.836054] 1082882 pages HighMem
[ 95.840099] 274370 pages reserved
[ 95.844136] 11446 pages shared
[ 95.848146] 410690 pages non-shared
[ 95.852142] ------------[ cut here ]------------
[ 95.856153] WARNING: at mm/oom_kill.c:327 __oom_kill_task+0x73/0xa0()
[ 95.860229] Hardware name: [ 95.864297] Modules linked in: s5h1411 tda18271 tda8290 tuner cx25840 snd_usb_audio snd_usb_lib snd_seq_midi snd_rawmidi snd_hwdep pvrusb2 dvb_core cx2341x v4l2_common pwc videodev tveeprom v4l1_compat
[ 95.873044] Pid: 2736, comm: xfsdump Not tainted 2.6.30 #1
[ 95.877428] Call Trace:
[ 95.881772] [<c01788c3>] ? __oom_kill_task+0x73/0xa0
[ 95.886159] [<c01788c3>] ? __oom_kill_task+0x73/0xa0
[ 95.890511] [<c01479f6>] ? warn_slowpath_common+0x76/0xd0
[ 95.894866] [<c01788c3>] ? __oom_kill_task+0x73/0xa0
[ 95.899220] [<c0147a63>] ? warn_slowpath_null+0x13/0x20
[ 95.903575] [<c01788c3>] ? __oom_kill_task+0x73/0xa0
[ 95.907932] [<c0178a01>] ? oom_kill_process+0x41/0x240
[ 95.912290] [<c017900d>] ? badness+0x17d/0x2b0
[ 95.916643] [<c0179284>] ? __out_of_memory+0x144/0x170
[ 95.921017] [<c0179304>] ? out_of_memory+0x54/0xb0
[ 95.925383] [<c017c142>] ? __alloc_pages_internal+0x442/0x480
[ 95.929781] [<c019b032>] ? cache_alloc_refill+0x2b2/0x500
[ 95.934179] [<c019b349>] ? __kmalloc+0xc9/0xd0
[ 95.938564] [<c02d6e39>] ? kmem_alloc+0x59/0xf0
[ 95.942949] [<c02d6f51>] ? kmem_zalloc+0x11/0x40
[ 95.947314] [<c029b56a>] ? xfs_getbmap+0x19a/0x7a0
[ 95.951668] [<c02928ab>] ? xfs_bmap_search_multi_extents+0x7b/0x100
[ 95.956056] [<c0292997>] ? xfs_bmap_search_extents+0x67/0x120
[ 95.960430] [<c04b3789>] ? io_schedule+0x29/0x40
[ 95.964810] [<c02dbfd0>] ? xfs_getbmapx_format+0x0/0x30
[ 95.969197] [<c04b3b0b>] ? __wait_on_bit_lock+0x5b/0x80
[ 95.973593] [<c01a01bc>] ? file_move+0x1c/0x60
[ 95.977969] [<c019d651>] ? __dentry_open+0xf1/0x2c0
[ 95.982370] [<c02dca7e>] ? xfs_ioc_getbmapx+0x6e/0x90
[ 95.986782] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 95.991225] [<c02ddce2>] ? xfs_file_ioctl+0x6c2/0x7d0
[ 95.995674] [<c02e0111>] ? xfs_read+0x171/0x290
[ 96.000107] [<c02b50f0>] ? xfs_iunlock+0x70/0xb0
[ 96.004569] [<c02d4192>] ? xfs_free_eofblocks+0x122/0x310
[ 96.009042] [<c02dd620>] ? xfs_file_ioctl+0x0/0x7d0
[ 96.013496] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 96.017988] [<c01abf8b>] ? vfs_ioctl+0x2b/0x90
[ 96.022449] [<c01ac06b>] ? do_vfs_ioctl+0x7b/0x590
[ 96.026867] [<c02f6f67>] ? _atomic_dec_and_lock+0x47/0x70
[ 96.031257] [<c01af415>] ? dput+0xd5/0x120
[ 96.035513] [<c01b54a8>] ? mntput_no_expire+0x18/0x110
[ 96.039642] [<c01ac5bd>] ? sys_ioctl+0x3d/0x70
[ 96.043751] [<c01220a1>] ? syscall_call+0x7/0xb
[ 96.047727] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 96.051573] [<c0120000>] ? vmx_create_vcpu+0x100/0x800
[ 96.055393] ---[ end trace 05f373a7e53c7489 ]---
[ 96.059088] tried to kill an mm-less task!

--

I re-ran the xfsdump without having any modules loaded:

Then I ran xfsdump (again):
xfsdump -e -L title -M title -f /nfs/path/root.img

http://home.comcast.net/~jpiszcz/20090611/dmesg2.txt

dmesg:
[ 1.388253] sd 1:0:0:0: [sdb] 1465149168 512-byte hardware sectors: (750 GB/698 GiB)
[ 1.392302] sd 1:0:0:0: [sdb] Write Protect is off
[ 1.396317] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 1.396329] scsi 2:0:0:0: CD-ROM TSSTcorp CD/DVDW SH-S183L SB02 PQ: 0 ANSI: 5
[ 1.396331] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.396373] sdb: sdb1
[ 1.408771] sd 1:0:0:0: [sdb] Attached SCSI disk
[ 1.418982] sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
[ 1.423106] Uniform CD-ROM driver Revision: 3.20
[ 1.427232] sr 2:0:0:0: Attached scsi CD-ROM sr0
[ 1.610651] UDF-fs: No partition found (1)
[ 1.616011] XFS mounting filesystem sda2
[ 1.629087] usb 2-1: configuration #1 chosen from 1 choice
[ 1.677313] Ending clean XFS mount for filesystem: sda2
[ 1.677344] VFS: Mounted root (xfs filesystem) readonly on device 8:2.
[ 1.681438] Freeing unused kernel memory: 316k freed
[ 1.843013] usb 2-2: new low speed USB device using uhci_hcd and address 3
[ 2.017021] usb 2-2: configuration #1 chosen from 1 choice
[ 2.039154] input: Microsoft Microsoft 5-Button Mouse with IntelliEye(TM) as /class/input/input4
[ 2.043605] generic-usb 0003:045E:0039.0001: input: USB HID v1.10 Mouse [Microsoft Microsoft 5-Button Mouse with IntelliEye(TM)] on usb-0000:00:1d.0-2/input0
[ 2.255013] usb 3-2: new low speed USB device using uhci_hcd and address 2
[ 2.425812] usb 3-2: configuration #1 chosen from 1 choice
[ 2.705758] generic-usb 0003:050D:1100.0002: hiddev96: USB HID v1.11 Device [Belkin Belkin UPS] on usb-0000:00:1d.1-2/input0
[ 2.959805] Adding 2056280k swap on /dev/sda1. Priority:-1 extents:1 across:2056280k SS
[ 4.053399] XFS mounting filesystem sdb1
[ 4.173486] Ending clean XFS mount for filesystem: sdb1
[ 4.472295] e1000e 0000:03:00.0: irq 30 for MSI/MSI-X
[ 4.524118] e1000e 0000:03:00.0: irq 30 for MSI/MSI-X
[ 7.468987] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[ 123.857734] xfsdump invoked oom-killer: gfp_mask=0x2d0, order=3, oomkilladj=0
[ 123.861449] Pid: 2491, comm: xfsdump Not tainted 2.6.30 #1
[ 123.865191] Call Trace:
[ 123.868812] [<c0178a6c>] ? oom_kill_process+0xac/0x240
[ 123.872490] [<c017900d>] ? badness+0x17d/0x2b0
[ 123.875995] [<c0179284>] ? __out_of_memory+0x144/0x170
[ 123.879524] [<c0179304>] ? out_of_memory+0x54/0xb0
[ 123.883047] [<c017c142>] ? __alloc_pages_internal+0x442/0x480
[ 123.886622] [<c019b032>] ? cache_alloc_refill+0x2b2/0x500
[ 123.890202] [<c019b349>] ? __kmalloc+0xc9/0xd0
[ 123.893771] [<c02d6e39>] ? kmem_alloc+0x59/0xf0
[ 123.897252] [<c02d6f51>] ? kmem_zalloc+0x11/0x40
[ 123.900643] [<c029b56a>] ? xfs_getbmap+0x19a/0x7a0
[ 123.904044] [<c02928ab>] ? xfs_bmap_search_multi_extents+0x7b/0x100
[ 123.907483] [<c0292997>] ? xfs_bmap_search_extents+0x67/0x120
[ 123.910914] [<c04b3789>] ? io_schedule+0x29/0x40
[ 123.914354] [<c02dbfd0>] ? xfs_getbmapx_format+0x0/0x30
[ 123.917778] [<c04b3b0b>] ? __wait_on_bit_lock+0x5b/0x80
[ 123.921182] [<c01a01bc>] ? file_move+0x1c/0x60
[ 123.924574] [<c019d651>] ? __dentry_open+0xf1/0x2c0
[ 123.927952] [<c02dca7e>] ? xfs_ioc_getbmapx+0x6e/0x90
[ 123.931325] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 123.934679] [<c02ddce2>] ? xfs_file_ioctl+0x6c2/0x7d0
[ 123.938040] [<c02e0111>] ? xfs_read+0x171/0x290
[ 123.941421] [<c02b50f0>] ? xfs_iunlock+0x70/0xb0
[ 123.944810] [<c02d4192>] ? xfs_free_eofblocks+0x122/0x310
[ 123.948223] [<c02dd620>] ? xfs_file_ioctl+0x0/0x7d0
[ 123.951624] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 123.955045] [<c01abf8b>] ? vfs_ioctl+0x2b/0x90
[ 123.958454] [<c01ac06b>] ? do_vfs_ioctl+0x7b/0x590
[ 123.961841] [<c02f6f67>] ? _atomic_dec_and_lock+0x47/0x70
[ 123.965221] [<c01af415>] ? dput+0xd5/0x120
[ 123.968596] [<c01b54a8>] ? mntput_no_expire+0x18/0x110
[ 123.971995] [<c01ac5bd>] ? sys_ioctl+0x3d/0x70
[ 123.975370] [<c01220a1>] ? syscall_call+0x7/0xb
[ 123.978721] [<c0205838>] ? journal_destroy+0x148/0x1e0
[ 123.982087] [<c0120000>] ? vmx_create_vcpu+0x100/0x800
[ 123.985442] Mem-Info:
[ 123.988765] DMA per-cpu:
[ 123.992064] CPU 0: hi: 0, btch: 1 usd: 0
[ 123.995385] CPU 1: hi: 0, btch: 1 usd: 0
[ 123.998622] Normal per-cpu:
[ 124.001779] CPU 0: hi: 186, btch: 31 usd: 0
[ 124.004945] CPU 1: hi: 186, btch: 31 usd: 25
[ 124.008051] HighMem per-cpu:
[ 124.011138] CPU 0: hi: 186, btch: 31 usd: 0
[ 124.014257] CPU 1: hi: 186, btch: 31 usd: 58
[ 124.017336] Active_anon:3977 active_file:1686 inactive_anon:822
[ 124.017337] inactive_file:367773 unevictable:0 dirty:444 writeback:0 unstable:1283
[ 124.017338] free:458376 slab:200088 mapped:3121 pagetables:272 bounce:0
[ 124.026921] DMA free:3780kB min:284kB low:352kB high:424kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:0 all_unreclaimable? no
[ 124.033860] lowmem_reserve[]: 0 867 4038 4038
[ 124.037351] Normal free:46460kB min:16096kB low:20120kB high:24144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:887976kB pages_scanned:0 all_unreclaimable? no
[ 124.044760] lowmem_reserve[]: 0 0 25374 25374
[ 124.048491] HighMem free:1783264kB min:512kB low:15228kB high:29948kB active_anon:15908kB inactive_anon:3288kB active_file:6744kB inactive_file:1471092kB unevictable:0kB present:3247956kB pages_scanned:0 all_unreclaimable? no
[ 124.056476] lowmem_reserve[]: 0 0 0 0
[ 124.060497] DMA: 9*4kB 12*8kB 12*16kB 4*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3780kB
[ 124.064676] Normal: 3898*4kB 1834*8kB 885*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 46472kB
[ 124.068914] HighMem: 872*4kB 3147*8kB 7562*16kB 8323*32kB 5774*64kB 2973*128kB 107*256kB 2*512kB 1*1024kB 3*2048kB 142*4096kB = 1783288kB
[ 124.073250] 369489 total pagecache pages
[ 124.077516] 0 pages in swap cache
[ 124.081697] Swap cache stats: add 0, delete 0, find 0/0
[ 124.085833] Free swap = 2056280kB
[ 124.089939] Total swap = 2056280kB
[ 124.102973] 1310720 pages RAM
[ 124.107035] 1082882 pages HighMem
[ 124.111095] 274370 pages reserved
[ 124.115123] 13146 pages shared
[ 124.119127] 407410 pages non-shared
[ 124.123114] Out of memory: kill process 2393 (sshd) score 2839 or a child
[ 124.127189] Killed process 2395 (bash)

Any clues?

Justin.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/