Re: Strange system hangs
From: Krzysztof Oledzki
Date: Tue Oct 16 2007 - 03:55:05 EST
On Sat, 29 Sep 2007, Nick Piggin wrote:
On Friday 28 September 2007 18:42, Krzysztof Oledzki wrote:
Hello,
I am experiencing weird system hangs. Once about 2-5 weeks system freezes
and stops accepting remote connections, so it is no longer possible to
connect to most important services: smtp (postfix), www (squid) or even
ssh. Such connection is accepted but then it hangs.
What is strange, that previously established ssh session is usable. It is
possible to work on such system until you do something stupid like "less
/var/log/all.log". Using strace I found that process blocks on:
Is this a regression? If so, what's the most recent kernel that didn't show
the problem?
The symptoms could be consistent with some place doing a
balance_dirty_pages while holding a lock that is required for IO, but I can't
see a smoking gun (you've got contention on i_mutex, but that should be
OK).
OK. It did it again :(, so I send additional debug information. Today I
discovered that "echo i > /proc/sysrq-trigger" makes my system being
usable.
Can you see if there is any memory under writeback that isn't being
completed (sysrq+M),
SysRq : Show Memory
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 11 Cold: hi: 62, btch: 15 usd: 8
CPU 1: Hot: hi: 186, btch: 31 usd: 170 Cold: hi: 62, btch: 15 usd: 23
Active:281859 inactive:140551 dirty:46161 writeback:0 unstable:0
free:39081 slab:50571 mapped:3261 pagetables:356 bounce:0
DMA free:8704kB min:512kB low:640kB high:768kB active:48kB inactive:0kB present:16256kB pages_scanned:12 all_unreclaimable? no
lowmem_reserve[]: 0 2015
Normal free:147620kB min:65016kB low:81268kB high:97524kB active:1127388kB inactive:562204kB present:2064260kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0
DMA: 4*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8704kB
Normal: 15781*4kB 522*8kB 264*16kB 56*32kB 9*64kB 8*128kB 6*256kB 1*512kB 1*1024kB 2*2048kB 16*4096kB = 147620kB
Swap cache: add 5022, delete 4928, find 2281/2804, race 0+0
Free swap = 2927208kB
Total swap = 2927608kB
Free swap: 2927208kB
524224 pages of RAM
0 pages of HIGHMEM
6345 reserved pages
249411 pages shared
94 pages swap cached
46161 pages dirty
0 pages writeback
3261 pages mapped
50571 pages slab
356 pages pagetables
also a list the locks held after the hang might be
helpful (compile in lockdep and sysrq+D)
SysRq : Show Locks Held
Showing all locks held in the system:
1 lock held by syslogd/2197:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by freshclam/2898:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by crond/3061:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by snmpd/3546:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by squid/3606:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by zabbix_agentd/3728:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by agetty/3791:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3793:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3794:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3795:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3796:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3797:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3798:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
1 lock held by agetty/3799:
#0: (&tty->atomic_read_lock){--..}, at: [<7829cdde>] read_chan+0x1b1/0x54b
2 locks held by bash/3887:
#0: (sysrq_key_table_lock){....}, at: [<782a6ea9>] __handle_sysrq+0x17/0x109
#1: (tasklist_lock){..-?}, at: [<7813d883>] debug_show_all_locks+0x3b/0x12d
1 lock held by amavisd/7227:
#0: (&mm->mmap_sem){----}, at: [<781196ad>] do_page_fault+0x152/0x535
1 lock held by sadc/8202:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by emerge/8204:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by mkpir/8297:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
1 lock held by screen/8365:
#0: (&inode->i_mutex){--..}, at: [<7814aabc>] generic_file_aio_write+0x43/0xb6
=============================================
Is anything currently running? (sysrq+P
Empty.
and even a full sysrq+T task list
could be useful).
SysRq : Show State
free sibling
task PC stack pid father child younger older
init S 7A1C1B4C 0 1 0 (NOTLB)
7a1c1b60 00000016 00000002 7a1c1b4c 7a1c1b48 00000000 7840d6e3 7a1d2000
0000000a 00000001 f7fbf490 f7fbe030 9c45a103 0001ff61 00005b8a f7fbf59c
7a022980 f71bdd00 219904e6 00000003 00000000 00000000 7a1c1b70 2199186c
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7812cada>] process_timeout+0x0/0x5
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7837cb10>] dev_queue_xmit+0x10b/0x213
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78172a90>] __d_lookup+0xbb/0x10b
[<78172a3f>] __d_lookup+0x6a/0x10b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78172ac3>] __d_lookup+0xee/0x10b
[<7816a323>] do_lookup+0x4f/0x143
[<78171ebc>] dput+0x16/0xe4
[<7816be16>] __link_path_walk+0xa1a/0xb5e
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7816c003>] link_path_walk+0xa9/0xb3
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78166c60>] cp_new_stat64+0xfc/0x10e
[<7816f30a>] sys_select+0xa0/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
kthreadd S F7FBEA60 0 2 0 (L-TLB)
7a1c3fd0 00000016 7851aa94 f7fbea60 00000001 786b7ec8 f7fbef90 f7fbef90
0000000a 00000000 f7fbea60 edcd0030 e200f0d9 0001df92 00000d92 f7fbeb6c
7a01a980 f7cafa40 00001890 00000246 7851aa80 00001890 7851aa60 00001890
Call Trace:
[<78134f4d>] kthreadd+0x61/0xfe
[<78134eec>] kthreadd+0x0/0xfe
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
migration/0 S 7A022980 0 3 2 (L-TLB)
7a1c9fa4 00000002 7811e4eb 7a022980 f7c7d550 7a1c9fa4 7811f199 7a1c7a00
00000001 00000000 7a1c74d0 f7e84b60 78531ca2 0001ff5b 00001431 7a1c75dc
7a01a980 e913a340 7a0229e8 00000046 7a01a980 7a01a980 7a01b2f4 7a01a980
Call Trace:
[<7811e4eb>] resched_task+0x3b/0x59
[<7811f199>] move_tasks+0x1f5/0x2bd
[<781221cf>] migration_thread+0x14a/0x1cd
[<78122085>] migration_thread+0x0/0x1cd
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksoftirqd/0 S 7A1CBFB4 0 4 2 (L-TLB)
7a1cbfc8 00000002 00000002 7a1cbfb4 7a1cbfb0 00000000 edcd1490 edcd1490
00000009 00000000 7a1c6aa0 785143e0 f1a2fd1e 0001e9af 0000065e 7a1c6bac
7a01a980 c053d340 202bebb7 00000003 00000000 00000000 7a018100 00000000
Call Trace:
[<78129a29>] ksoftirqd+0x55/0x134
[<781299d4>] ksoftirqd+0x0/0x134
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
watchdog/0 S 7A01A980 0 5 2 (L-TLB)
7a1cdfcc 00000006 7840d6e3 7a01a980 7a1cdfcc 7813e99c 00000002 00000046
00000001 00000000 7a1c6070 f7fbf490 35f147b4 0000000a 00002c3f 7a1c617c
7a01a980 78514180 7a1c6530 ffffffff 78596a0c 01a85000 00000000 78146910
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78146910>] watchdog+0x0/0x58
[<78146958>] watchdog+0x48/0x58
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
migration/1 S 7A1D1F90 0 6 2 (L-TLB)
7a1d1fa4 00000002 00000002 7a1d1f90 7a1d1f8c 00000000 7a1cfa40 7a1cfa40
00000001 00000001 7a1cf510 f7fbe030 04d4c284 0001ff60 00001b5e 7a1cf61c
7a022980 e8d34580 2198ea1b 00000003 00000000 00000000 7a0232f4 7a022980
Call Trace:
[<781221cf>] migration_thread+0x14a/0x1cd
[<78122085>] migration_thread+0x0/0x1cd
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksoftirqd/1 S 7A1D5FB4 0 7 2 (L-TLB)
7a1d5fc8 00000002 00000002 7a1d5fb4 7a1d5fb0 00000000 00000046 00000000
0000000a 00000001 7a1ceae0 f7fbe030 07b31f2a 0001ff3a 0004c80b 7a1cebec
7a022980 e913a340 21966ac7 00000003 00000000 00000000 7a020100 00000001
Call Trace:
[<78129a29>] ksoftirqd+0x55/0x134
[<781299d4>] ksoftirqd+0x0/0x134
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
watchdog/1 S 7A022980 0 8 2 (L-TLB)
7a1d7fcc 00000006 7840d6e3 7a022980 7a1d7fcc 7813e99c 00000002 00000046
00000001 00000001 7a1ce0b0 7a1ceae0 372914ac 0000000a 00003a96 7a1ce1bc
7a022980 78514180 7a1ce570 ffffffff 78596a0c 01a8d000 00000001 78146910
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78146910>] watchdog+0x0/0x58
[<78146958>] watchdog+0x48/0x58
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
events/0 S F7FBDFA0 0 9 2 (L-TLB)
f7fbdfb4 00000086 00000002 f7fbdfa0 f7fbdf9c 00000000 f7ea1a80 00000000
0000000a 00000000 f7ea1550 785143e0 959db3cd 0001ff61 0000158b f7ea165c
7a01a980 c053d840 21990475 00000003 00000000 00000000 f7f940c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
events/1 S F7FB9FA0 0 10 2 (L-TLB)
f7fb9fb4 00000086 00000002 f7fb9fa0 f7fb9f9c 00000000 f7ea1050 00000000
0000000a 00000001 f7ea0b20 f7fbe030 992e5f8c 0001ff61 000013bc f7ea0c2c
7a022980 e913a340 219904b2 00000003 00000000 00000000 f7c4df40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
khelper S 00000000 0 11 2 (L-TLB)
f7fbbfb4 00000086 00000046 00000000 00000000 f7c4defc f7ea0620 00000000
00000009 00000001 f7ea00f0 f7cc8130 6fcbad27 0001df91 000011c4 f7ea01fc
7a022980 f6b15840 00000046 f7c4dee8 00000286 f7c4dee8 f7c4dec0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kblockd/0 S F7E3BFA0 0 64 2 (L-TLB)
f7e3bfb4 00000086 00000002 f7e3bfa0 f7e3bf9c 00000000 f7e85090 00000000
0000000a 00000000 f7e84b60 785143e0 6ff1ad71 0001ff61 00001779 f7e84c6c
7a01a980 e913a340 219901fd 00000003 00000000 00000000 f7e951c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kblockd/1 S F7E3DFA0 0 65 2 (L-TLB)
f7e3dfb4 00000086 00000002 f7e3dfa0 f7e3df9c 00000000 00000000 00000000
0000000a 00000001 f7e85590 f7fbe030 593eee3f 0001ff61 00001932 f7e8569c
7a022980 d3e4acc0 2199007e 00000002 00000000 00000000 f7e95140 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kacpid S 00000000 0 66 2 (L-TLB)
f7e3ffb4 00000086 00000046 00000000 00000000 f7c76cfc f7c7c620 00000000
00000001 00000000 f7c7c0f0 f7fbf490 3bbce431 0000000a 0000072f f7c7c1fc
7a01a980 78514180 00000046 f7c76ce8 00000286 f7c76ce8 f7c76cc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kacpi_notify S 00000000 0 67 2 (L-TLB)
f7e41fb4 00000086 00000046 00000000 00000000 f7c76bfc f7c7d050 00000000
00000001 00000000 f7c7cb20 f7fbf490 3bbd4590 0000000a 00000720 f7c7cc2c
7a01a980 78514180 00000046 f7c76be8 00000286 f7c76be8 f7c76bc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ata/0 S 00000000 0 147 2 (L-TLB)
f7e21fb4 00000086 00000046 00000000 00000000 f7c4bd7c f7e23a40 00000000
0000000a 00000000 f7e23510 f7cd0030 a23c198a 0000000b 0002d0fb f7e2361c
7a01a980 78514180 00000046 f7c4bd68 00000286 f7c4bd68 f7c4bd40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ata/1 S 00000000 0 148 2 (L-TLB)
f7e19fb4 00000086 00000046 00000000 00000000 f7c4bcfc f7e2a620 00000000
0000000a 00000001 f7e2a0f0 f7fbf490 7e41e2a8 0000000b 00005ba5 f7e2a1fc
7a022980 78514180 00000046 f7c4bce8 00000286 f7c4bce8 f7c4bcc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ata_aux S F7E43FA0 0 149 2 (L-TLB)
f7e43fb4 00000086 00000002 f7e43fa0 f7e43f9c 00000000 f7e1a5a0 00000000
00000001 00000000 f7e1a070 785143e0 3dbcad9d 0000000a 00000e19 f7e1a17c
7a01a980 78514180 fffb6d29 00000003 00000000 00000000 f7c4bc40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksuspend_usbd S F7E29FA0 0 150 2 (L-TLB)
f7e29fb4 00000086 00000002 f7e29fa0 f7e29f9c 00000000 f7e84660 00000000
00000001 00000000 f7e84130 785143e0 3dbd4cb7 0000000a 00001042 f7e8423c
7a01a980 78514180 fffb6d29 00000003 00000000 00000000 f7c4bb40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
khubd S 00000000 0 153 2 (L-TLB)
f7e05f54 00000086 00000046 00000000 00000000 78538a34 f7e225e0 00000000
0000000a 00000000 f7e220b0 f7fbf490 d1ac4454 0000000b 00005717 f7e221bc
7a01a980 78514180 00000046 78538a20 00000286 78538a20 00000000 f7d36200
Call Trace:
[<78316b2e>] hub_thread+0x98f/0x9f9
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7840ae59>] __sched_text_start+0x701/0x810
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7831619f>] hub_thread+0x0/0x9f9
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kseriod S 00000000 0 155 2 (L-TLB)
7a1fdf9c 00000002 00000046 00000000 00000000 7853bcd4 f7e0a660 00000000
0000000a 00000000 f7e0a130 f7fbf490 0dfac2e3 0000000c 00003730 f7e0a23c
7a01a980 78514180 00000046 7853bcc0 00000216 7853bcc0 f7d50580 7832efe5
Call Trace:
[<7832efe5>] serio_thread+0x0/0x2bb
[<7832f255>] serio_thread+0x270/0x2bb
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7832efe5>] serio_thread+0x0/0x2bb
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kswapd0 S F7E59F44 0 187 2 (L-TLB)
f7e59f58 00000086 00000002 f7e59f44 f7e59f40 00000000 7a1fb010 00000000
00000009 00000001 7a1faae0 f7fbe030 1a81a860 0001e6a0 00e7fbcc 7a1fabec
7a022980 abc7dd40 1ff863fd 00000003 00000000 00000000 00000001 ffffffff
Call Trace:
[<78150a2e>] kswapd+0xb0/0x40f
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7815097e>] kswapd+0x0/0x40f
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
aio/0 S 00000000 0 188 2 (L-TLB)
f7e5dfb4 00000086 00000046 00000000 00000000 f7ed547c 7a1fba40 00000000
00000001 00000000 7a1fb510 f7fbea60 42056d94 0000000a 00000dba 7a1fb61c
7a01a980 78514180 00000046 f7ed5468 00000286 f7ed5468 f7ed5440 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
aio/1 S 00000000 0 189 2 (L-TLB)
f7e5ffb4 00000086 00000046 00000000 00000000 f7ed53fc 7a1f25a0 00000000
00000001 00000001 7a1f2070 f7fbf490 4205dc87 0000000a 000009cd 7a1f217c
7a022980 78514180 00000046 f7ed53e8 00000286 f7ed53e8 f7ed53c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
cryptd S F7E7FFA8 0 190 2 (L-TLB)
f7e7ffbc 00000082 00000002 f7e7ffa8 f7e7ffa4 00000000 787ce0d8 f7fb5590
00000001 00000000 f7fb5590 785143e0 42116cfc 0000000a 000076c7 f7fb569c
7a01a980 78514180 fffb6d71 00000003 00000000 00000000 00000000 00000000
Call Trace:
[<7820a221>] cryptd_thread+0x95/0xa5
[<7820a18c>] cryptd_thread+0x0/0xa5
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_0 S 7A1EA030 0 320 2 (L-TLB)
f7ca1f80 00000092 f7e0f058 7a1ea030 7840d6e3 f7e0f000 f7cef000 7813e99c
0000000a 00000000 7a1ea030 f7fbf490 7ebe3537 0000000b 00005ddb 7a1ea13c
7a01a980 78514180 00000292 f7cef000 f7e0f000 00000000 00000282 f7e0f000
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_1 S F7CA5F6C 0 322 2 (L-TLB)
f7ca5f80 00000092 00000002 f7ca5f6c f7ca5f68 00000000 00000000 7813e99c
00000003 00000000 f7cf94d0 785143e0 7e3733ad 0000000b 0000c8a9 f7cf95dc
7a01a980 78514180 fffb8236 00000003 00000000 00000000 00000282 f7cbd800
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_2 S F7E0B590 0 339 2 (L-TLB)
f7ca7f80 00000092 7a1d8058 f7e0b590 7840d6e3 7a1d8000 00000000 7813e99c
00000006 00000000 f7e0b590 f7fbf490 9835b106 0000000b 004284a4 f7e0b69c
7a01a980 78514180 00000292 00000282 7a1d8000 00000000 00000282 7a1d8000
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
scsi_eh_3 S F7CD0030 0 341 2 (L-TLB)
f7ca3f80 00000092 f7cbd058 f7cd0030 7840d6e3 f7cbd000 00000000 7813e99c
00000008 00000000 f7cd0030 f7fbf490 a28957bc 0000000b 003e72b9 f7cd013c
7a01a980 78514180 00000292 00000282 f7cbd000 00000000 00000282 f7cbd000
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<782e9a32>] scsi_error_handler+0x3b/0x28d
[<782e99f7>] scsi_error_handler+0x0/0x28d
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kpsmoused S 00000000 0 407 2 (L-TLB)
f7d75fb4 00000086 00000046 00000000 00000000 f7d1767c f7d005e0 00000000
00000009 00000000 f7d000b0 f7fbf490 d3b0a6af 0000000b 0000087a f7d001bc
7a01a980 78514180 00000046 f7d17668 00000286 f7d17668 f7d17640 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kcryptd/0 S 00000000 0 412 2 (L-TLB)
f7d77fb4 00000086 00000046 00000000 00000000 f7d170fc f7e33090 00000000
00000009 00000000 f7e32b60 f7fbf490 e6cc0d2f 0000000b 000007f4 f7e32c6c
7a01a980 78514180 00000046 f7d170e8 00000286 f7d170e8 f7d170c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kcryptd/1 S F7D71FA0 0 413 2 (L-TLB)
f7d71fb4 00000086 00000002 f7d71fa0 f7d71f9c 00000000 f7d01010 00000000
00000008 00000001 f7d00ae0 f7fbe030 e6cc9219 0000000b 000010ad f7d00bec
7a022980 78514180 fffb8914 00000003 00000000 00000000 f7d5cf40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kmpathd/0 S 00000000 0 414 2 (L-TLB)
f7d73fb4 00000086 00000046 00000000 00000000 f7d5cd7c f7cf85a0 00000000
00000009 00000000 f7cf8070 f7fbf490 e6cd348c 0000000b 00000786 f7cf817c
7a01a980 78514180 00000046 f7d5cd68 00000286 f7d5cd68 f7d5cd40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kmpathd/1 S F7D6DFA0 0 415 2 (L-TLB)
f7d6dfb4 00000086 00000002 f7d6dfa0 f7d6df9c 00000000 f7cf8fd0 00000000
00000008 00000001 f7cf8aa0 f7fbe030 e6cda902 0000000b 00000f2f f7cf8bac
7a022980 78514180 fffb8914 00000003 00000000 00000000 f7d5ccc0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ksnapd S 00000000 0 416 2 (L-TLB)
f7d31fb4 00000086 00000046 00000000 00000000 f7d5c9fc f7d01a40 00000000
00000008 00000000 f7d01510 f7fbf490 e72c4b66 0000000b 0000070f f7d0161c
7a01a980 78514180 00000046 f7d5c9e8 00000286 f7d5c9e8 f7d5c9c0 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kedac S F7D7BF80 0 419 2 (L-TLB)
f7d7bf94 00000086 00000002 f7d7bf80 f7d7bf7c 00000000 7840d6e3 7a1d2000
0000000a 00000001 f7e0ab60 f7fbe030 acabac43 0001ff61 00000d5e f7e0ac6c
7a022980 e913a340 219905f9 00000003 00000000 00000000 f7d7bfa4 219909e0
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<78363dd9>] edac_kernel_thread+0x0/0xf3
[<7812cada>] process_timeout+0x0/0x5
[<78363dd9>] edac_kernel_thread+0x0/0xf3
[<78363ea0>] edac_kernel_thread+0xc7/0xf3
[<78363dd9>] edac_kernel_thread+0x0/0xf3
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
md4_raid5 S F7E2AB20 0 434 2 (L-TLB)
7a255f84 00000006 7a21a920 f7e2ab20 7840d727 00000000 00000000 7a224d5c
0000000a 00000000 f7e2ab20 f6928030 63aadcb9 0001ff61 00002f44 f7e2ac2c
7a01a980 e913a340 00000000 00000000 f7e2ab20 f7e2b050 7fffffff 7fffffff
Call Trace:
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<78354f55>] md_thread+0xab/0xdc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78354eaa>] md_thread+0x0/0xdc
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
md15_raid10 S 7A1F3A00 0 437 2 (L-TLB)
7a253f84 00000006 7a1f34d0 7a1f3a00 785cde94 7a1f34d0 7840d6e3 785cde80
0000000a 00000000 7a1f34d0 f6928a60 39bb729c 0001ff61 000011b8 7a1f35dc
7a01a980 f58bf040 785cde80 7812ccc1 00000000 00000216 7a253f94 219911f1
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7812cada>] process_timeout+0x0/0x5
[<78354f55>] md_thread+0xab/0xdc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78354eaa>] md_thread+0x0/0xdc
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
md0_raid1 S F7C7DA80 0 439 2 (L-TLB)
f719ff84 00000086 f7c7d550 f7c7da80 785cde94 f7c7d550 7840d6e3 785cde80
0000000a 00000000 f7c7d550 f6928030 706bbe1a 0001ff61 00002cc3 f7c7d65c
7a01a980 e913a340 785cde80 7812ccc1 00000000 00000296 f719ff94 2199158a
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7812cada>] process_timeout+0x0/0x5
[<78354f55>] md_thread+0xab/0xdc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78354eaa>] md_thread+0x0/0xdc
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 441 2 (L-TLB)
f717dfa8 00000082 00000046 00000000 00000000 f7199d3c f7c74608 f7c745e0
0000000a 00000000 f7c740b0 f6928030 575613ff 0001ff61 000092f3 f7c741bc
7a01a980 f6e31800 00000046 00000246 f7199c14 f7199c00 f7199c14 f7199c00
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S F6DD5F94 0 584 2 (L-TLB)
f6dd5fa8 00000082 00000002 f6dd5f94 f6dd5f90 00000000 f7fb4688 f7fb4660
0000000a 00000001 f7fb4130 f7fbe030 545ba352 0001ff61 0000700b f7fb423c
7a022980 e632d2c0 2199002b 00000003 00000000 00000000 f68c8814 f68c8800
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S F588DF94 0 585 2 (L-TLB)
f588dfa8 00000082 00000002 f588df94 f588df90 00000000 f7cd19e8 f7cd19c0
0000000a 00000000 f7cd1490 785143e0 f0b5365d 0001ff24 00001614 f7cd159c
7a01a980 f7cafa40 219507da 00000003 00000000 00000000 f58c1814 f58c1800
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 586 2 (L-TLB)
f6fdbfa8 00000082 00000046 00000000 00000000 f58c113c 7a1dcfb8 7a1dcf90
00000009 00000000 7a1dca60 f7cb80b0 53c6eaba 0000000d 00001d05 7a1dcb6c
7a01a980 f68e9ac0 00000046 00000246 f58c1014 f58c1000 f58c1014 f58c1000
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 587 2 (L-TLB)
f587bfa8 00000082 00000046 00000000 00000000 f64c993c f7c6da28 f7c6da00
0000000a 00000001 f7c6d4d0 f7ea0b20 f44b1328 0001e632 0000149c f7c6d5dc
7a022980 e632d2c0 00000046 00000246 f64c9814 f64c9800 f64c9814 f64c9800
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
kjournald S 00000000 0 588 2 (L-TLB)
f58c9fa8 00000082 00000046 00000000 00000000 f64c913c 7a1dc588 7a1dc560
00000008 00000000 7a1dc030 f7cb80b0 5d289ec0 0000000d 00001bb0 7a1dc13c
7a01a980 f68e9ac0 00000046 00000246 f64c9014 f64c9000 f64c9014 f64c9000
Call Trace:
[<781aeb09>] kjournald+0x16d/0x1ec
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781ae99c>] kjournald+0x0/0x1ec
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
syslogd D 00000000 0 2197 1 (NOTLB)
f6267c74 00000082 00000000 00000000 7813e7a9 f7cc8130 7840d6e3 785cde80
00000007 00000000 f7cc8130 f7c74ae0 abd727e3 0001ff61 00002fdc f7cc823c
7a01a980 f6b15840 785cde80 7812ccc1 00000000 00000292 f6267c84 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<7816402f>] do_sync_readv_writev+0xc1/0xfe
[<78139494>] getnstimeofday+0x30/0xbc
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78163ee4>] rw_copy_check_uvector+0x50/0xaa
[<781646b4>] do_readv_writev+0x99/0x164
[<78199b90>] ext3_file_write+0x0/0x92
[<781647bc>] vfs_writev+0x3d/0x48
[<78164b9e>] sys_writev+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
klogd S F5F37DB8 0 2211 1 (NOTLB)
f5f37dcc 00000086 00000002 f5f37db8 f5f37db4 00000000 00000000 f6f77f54
00000007 00000000 f7c6caa0 785143e0 579dd14b 0001ff41 000c175e f7c6cbac
7a01a980 f62ada40 2196e5cc 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783cc30c>] unix_wait_for_peer+0x75/0x9a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
[<78370f09>] sock_aio_write+0xcb/0xd7
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781649e7>] vfs_read+0xcf/0x10a
[<781648aa>] vfs_write+0x9e/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
acpid S F5C35BE8 0 2265 1 (NOTLB)
f5c35bfc 00000082 00000002 f5c35be8 f5c35be4 00000000 78528e74 f7cc1550
00000001 00000000 f7cc1550 785143e0 87711dce 0000000f 0000efdb f7cc165c
7a01a980 f6b150c0 fffbc622 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<783cbaf6>] unix_poll+0x1a/0x9a
[<7816e77e>] do_sys_poll+0x258/0x327
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7840af41>] __sched_text_start+0x7e9/0x810
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<783cc862>] unix_dgram_sendmsg+0x3b3/0x431
[<7840d33a>] _read_lock+0x33/0x3e
[<7840d699>] _read_unlock+0x25/0x3b
[<783cc862>] unix_dgram_sendmsg+0x3b3/0x431
[<78371838>] sock_sendmsg+0xbc/0xd4
[<78171ebc>] dput+0x16/0xe4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<781383ba>] up_read+0x14/0x26
[<781197d3>] do_page_fault+0x278/0x535
[<78103e68>] restore_nocheck+0x12/0x15
[<7811955b>] do_page_fault+0x0/0x535
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816e881>] sys_poll+0x34/0x37
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
amavisd S F31C9B4C 0 2823 1 (NOTLB)
f31c9b60 00000096 00000002 f31c9b4c f31c9b48 00000000 7840d6e3 7a1d2000
00000007 00000001 f7e32130 f7fbe030 65ed4f6c 0001ff60 000151f2 f7e3223c
7a022980 f64f1800 2198f07d 00000003 00000000 00000000 f31c9b70 2199178d
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7812cada>] process_timeout+0x0/0x5
[<7816ec1d>] do_select+0x399/0x3e7
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7838b171>] __nf_ct_refresh_acct+0xcf/0x10a
[<7816f1be>] __pollwait+0x0/0xac
[<7838b171>] __nf_ct_refresh_acct+0xcf/0x10a
[<7838eaf9>] tcp_packet+0xa0f/0xa3e
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<783bb722>] inet_sock_destruct+0x179/0x1bb
[<783b147b>] tcp_v4_rcv+0x51a/0x787
[<783994b1>] ip_local_deliver+0x1bd/0x1c8
[<78398c10>] ip_local_deliver_finish+0x0/0x14c
[<783992bb>] ip_rcv+0x498/0x4d1
[<7839896c>] ip_rcv_finish+0x0/0x2a4
[<78139494>] getnstimeofday+0x30/0xbc
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<782be370>] e1000_clean_tx_irq+0xbc/0x2c5
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7812e2a3>] group_send_sig_info+0x12/0x56
[<7812e378>] kill_pid_info+0x61/0x7b
[<7812f183>] sys_kill+0x108/0x125
[<7816f30a>] sys_select+0xa0/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
clamd S 00000000 0 2889 1 (NOTLB)
f32d1e58 00000086 00000002 00000000 00010822 00000000 00000000 ee9a69ac
00000001 00000001 f7cb80b0 db4c8030 6f522ff7 0001e655 00010e05 f7cb81bc
7a022980 f68e9ac0 00000000 00000000 f7cb80b0 f7cb85e0 7fffffff f5e2d380
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<783787ab>] skb_recv_datagram+0x13f/0x1c4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783ccdf1>] unix_accept+0x51/0xde
[<7837262e>] sys_accept+0xb2/0x187
[<78166c60>] cp_new_stat64+0xfc/0x10e
[<781671ca>] sys_fstat64+0x1e/0x23
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
freshclam D 00000000 0 2898 1 (NOTLB)
f2f43cc4 00000082 00000000 00000000 7813e7a9 f6928030 7840d6e3 785cde80
00000008 00000000 f6928030 f7cc8130 abd6f807 0001ff61 00003332 f692813c
7a01a980 f58bfa40 785cde80 7812ccc1 00000000 00000296 f2f43cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
named S F0989D0C 0 2959 1 (NOTLB)
f0989d20 00200092 00000002 f0989d0c f0989d08 00000000 00000000 f6f77f54
0000000a 00000000 f7e13490 785143e0 0e816649 0001e659 00013c17 f7e1359c
7a01a980 f58bf2c0 1ff3b86f 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783cc30c>] unix_wait_for_peer+0x75/0x9a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
[<78371838>] sock_sendmsg+0xbc/0xd4
[<78171ebc>] dput+0x16/0xe4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78371b64>] sys_sendto+0x11b/0x13b
[<78169ed9>] path_release+0xa/0x20
[<78166c60>] cp_new_stat64+0xfc/0x10e
[<78371bbb>] sys_send+0x37/0x3b
[<78372830>] sys_socketcall+0x12d/0x242
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
crond D 00000000 0 3061 1 (NOTLB)
f1367cc4 00000082 00000000 00000000 7813e7a9 f7e02b20 7840d6e3 7a1d2000
00000008 00000001 f7e02b20 edcd0a60 abd708dc 0001ff61 0000367c f7e02c2c
7a022980 f58bfcc0 7a1d2000 7812ccc1 00000000 00000296 f1367cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
mdadm S F1ADDB4C 0 3110 1 (NOTLB)
f1addb60 00000096 00000002 f1addb4c f1addb48 00000000 7840d6e3 785cde80
00000007 00000000 f6928a60 785143e0 39bb937f 0001ff61 000020e3 f6928b6c
7a01a980 f58bf040 2198fe6a 00000003 00000000 00000000 f1addb70 2199e8c9
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7812cada>] process_timeout+0x0/0x5
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<7840af41>] __sched_text_start+0x7e9/0x810
[<7840ae59>] __sched_text_start+0x701/0x810
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7811e3ba>] __wake_up+0x32/0x43
[<7840b09e>] preempt_schedule+0x46/0x58
[<7834dc68>] md_wakeup_thread+0x26/0x28
[<78354e25>] md_ioctl+0x1261/0x12e6
[<7840d623>] _spin_unlock+0x25/0x3b
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7840bd93>] __mutex_unlock_slowpath+0x105/0x127
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78352e61>] md_open+0x51/0x58
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<781335c2>] call_rcu+0x64/0x69
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816f30a>] sys_select+0xa0/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
portmap S F1571BE8 0 3161 1 (NOTLB)
f1571bfc 00000082 00000002 f1571be8 f1571be4 00000000 00000000 00000000
00000007 00000001 7a1dd490 f7fbe030 e696d8aa 00000013 000281e5 7a1dd59c
7a022980 f6e31080 fffc0fae 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<783b435a>] udp_poll+0x10/0xdb
[<7816e77e>] do_sys_poll+0x258/0x327
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7840d1f4>] _spin_lock_bh+0x38/0x43
[<783b57a5>] udp_sendmsg+0x4a7/0x5b9
[<783ba7cb>] inet_sendmsg+0x3b/0x45
[<78371838>] sock_sendmsg+0xbc/0xd4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<78377ed6>] verify_iovec+0x3e/0x70
[<783719e4>] sys_sendmsg+0x194/0x1f9
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7814cb18>] free_hot_cold_page+0x13b/0x167
[<7814cb24>] free_hot_cold_page+0x147/0x167
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814cb58>] __pagevec_free+0x14/0x1a
[<7814ecc5>] release_pages+0x126/0x12e
[<78159060>] anon_vma_unlink+0x13/0x53
[<7840d623>] _spin_unlock+0x25/0x3b
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816e881>] sys_poll+0x34/0x37
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
rpciod/0 S 00000000 0 3216 2 (L-TLB)
ec88dfb4 00000086 00000046 00000000 00000000 f6343c7c 7a1eb9c0 00000000
0000000a 00000000 7a1eb490 89333550 78dbc731 0001e655 0000401b 7a1eb59c
7a01a980 c053d840 00000046 f6343c68 00000286 f6343c68 f6343c40 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
rpciod/1 S 00000000 0 3217 2 (L-TLB)
ec8e7fb4 00000086 00000046 00000000 00000000 f5a6867c f69299c0 00000000
0000000a 00000001 f6929490 a79ec0b0 2e2b55c5 00000834 0000ce3a f692959c
7a022980 a71d7340 00000046 f5a68668 00000286 f5a68668 f5a68640 78132a2b
Call Trace:
[<78132a2b>] worker_thread+0x0/0xc5
[<78132ab3>] worker_thread+0x88/0xc5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
lockd S 00000001 0 3218 2 (L-TLB)
ec865f18 00000082 7813534c 00000001 784d6071 f7e02628 00000000 00000000
00000001 00000000 f7e020f0 f7ca8a60 e69b3708 00000013 0000a1e7 f7e021fc
7a01a980 f68e90c0 f7e02620 f7e020f0 781299b4 f0fecd7c 7fffffff f0fec000
Call Trace:
[<7813534c>] add_wait_queue+0x12/0x30
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<783ffbab>] svc_recv+0x249/0x3ca
[<7812027d>] default_wake_function+0x0/0xc
[<781e6643>] lockd+0xe8/0x1fd
[<78121e77>] schedule_tail+0x0/0xb6
[<78103e68>] restore_nocheck+0x12/0x15
[<781e655b>] lockd+0x0/0x1fd
[<781e655b>] lockd+0x0/0x1fd
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
ntpd S EC959B4C 0 3269 1 (NOTLB)
ec959b60 00200096 00000002 ec959b4c ec959b48 00000000 f7cb05a0 00200046
0000000a 00000000 f7cb0070 785143e0 a27c1eef 0001ff61 0000aea1 f7cb017c
7a01a980 f7cafa40 2199054d 00000003 00000000 00000000 7fffffff ec959f9c
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783b435a>] udp_poll+0x10/0xdb
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<783760ad>] skb_dequeue+0x39/0x3f
[<783786ed>] skb_recv_datagram+0x81/0x1c4
[<783b5b8f>] udp_recvmsg+0x57/0x1cd
[<78372cc9>] sock_common_recvmsg+0x3e/0x54
[<78371763>] sock_recvmsg+0xcf/0xe8
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7837221d>] sys_recvmsg+0x11d/0x1cf
[<78371b64>] sys_sendto+0x11b/0x13b
[<7840d732>] _spin_unlock_irq+0x2b/0x41
[<78103986>] do_notify_resume+0x511/0x5f3
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sqlgrey S 00000000 0 3323 1 (NOTLB)
ec79fb60 00200096 786b7ed0 00000000 7837cbf7 00200046 00000000 00000000
00000004 00000001 f7cb0aa0 f7fbf490 b29583a6 0001f67f 00068968 f7cb0bac
7a022980 f68e90c0 7813e99c 00000000 00200046 ec975398 7fffffff ec79ff9c
Call Trace:
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<783b133f>] tcp_v4_rcv+0x3de/0x787
[<7840d623>] _spin_unlock+0x25/0x3b
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7839949e>] ip_local_deliver+0x1aa/0x1c8
[<78398c10>] ip_local_deliver_finish+0x0/0x14c
[<783992bb>] ip_rcv+0x498/0x4d1
[<7839896c>] ip_rcv_finish+0x0/0x2a4
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c73c>] process_backlog+0xef/0xfa
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c9d8>] net_rx_action+0x162/0x18f
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
smartd S EA68DD0C 0 3487 1 (NOTLB)
ea68dd20 00000092 00000002 ea68dd0c ea68dd08 00000000 00000000 f6f77f54
00000009 00000000 ec907490 785143e0 ad75944f 0001e66c 00034dc2 ec90759c
7a01a980 f6e11040 1ff502a1 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783cc30c>] unix_wait_for_peer+0x75/0x9a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783cc80c>] unix_dgram_sendmsg+0x35d/0x431
[<78371838>] sock_sendmsg+0xbc/0xd4
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813facb>] lock_release_non_nested+0xec/0x14d
[<783cdc63>] unix_dgram_connect+0x8d/0x157
[<783cdd00>] unix_dgram_connect+0x12a/0x157
[<783cdc63>] unix_dgram_connect+0x8d/0x157
[<78371b64>] sys_sendto+0x11b/0x13b
[<7840d623>] _spin_unlock+0x25/0x3b
[<783cdd00>] unix_dgram_connect+0x12a/0x157
[<7837135e>] sys_connect+0x72/0x9c
[<78371bbb>] sys_send+0x37/0x3b
[<78372830>] sys_socketcall+0x12d/0x242
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
snmpd D 00000000 0 3546 1 (NOTLB)
ea333cc4 00200082 00000000 00000000 7813e7a9 f7c74ae0 7840d6e3 785cde80
00000007 00000000 f7c74ae0 89333550 abd756a0 0001ff61 00002ebd f7c74bec
7a01a980 f6e31800 785cde80 7812ccc1 00000000 00200296 ea333cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squid S E9A41F20 0 3604 1 (NOTLB)
e9a41f34 00200082 00000002 e9a41f20 e9a41f1c 00000000 e9e6da40 e9e6da40
00000005 00000000 e9e6d510 785143e0 74bc2237 00000016 00014e53 e9e6d61c
7a01a980 f6e31d00 fffc3aae 00000003 00000000 00000000 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squid D 00000000 0 3606 3604 (NOTLB)
e9a65cc4 00000082 00000000 00000000 7813e7a9 f7cb9510 7840d6e3 7a1d2000
00000007 00000001 f7cb9510 edcd1490 abd7ba4d 0001ff61 000021db f7cb961c
7a022980 f6ce25c0 7a1d2000 7812ccc1 00000000 00000296 e9a65cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S E9239B4C 0 3663 1 (NOTLB)
e9239b60 00200096 00000002 e9239b4c e9239b48 00000000 00000000 00000000
00000006 00000000 e9b7e0f0 785143e0 61b30795 0001ff2d 000245bd e9b7e1fc
7a01a980 f62ad540 219595ea 00000003 00000000 00000000 7fffffff e9239f9c
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816e4d2>] free_poll_entry+0xe/0x16
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7840d623>] _spin_unlock+0x25/0x3b
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813e7a9>] mark_held_locks+0x39/0x53
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<782be370>] e1000_clean_tx_irq+0xbc/0x2c5
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d727>] _spin_unlock_irq+0x20/0x41
[<7816f340>] sys_select+0xd6/0x188
[<78103e68>] restore_nocheck+0x12/0x15
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S 7A1F2AA0 0 3722 1 (NOTLB)
e9369f34 00200082 7854ec14 7a1f2aa0 00000001 786b7e90 7a1f2fd0 7a1f2fd0
00000001 00000001 7a1f2aa0 f7fb4b60 0ec74ab6 00000017 0000b6ef 7a1f2bac
7a022980 e9314540 00000000 00200246 7854ec00 00000001 00000e8c 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<781197d3>] do_page_fault+0x278/0x535
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S E93D1F2C 0 3724 3722 (NOTLB)
e93d1f40 00200092 00000002 e93d1f2c e93d1f28 00000000 7813e99c 7a02391c
00000008 00000001 ec8de0f0 f7fbe030 aa5c0061 0001ff61 0004acea ec8de1fc
7a022980 e9314cc0 219905d2 00000003 00000000 00000000 e93d1f58 00000001
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840c5af>] do_nanosleep+0x48/0x73
[<78137d84>] hrtimer_nanosleep+0x39/0xdc
[<78137879>] hrtimer_wakeup+0x0/0x18
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78137e70>] sys_nanosleep+0x49/0x59
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S 00000070 0 3725 3722 (NOTLB)
e8ccbe44 00200086 00000001 00000070 7a01b3e0 00000002 00000000 ead5e140
00000001 00000000 e90b8b60 e9e6cae0 0f7701b8 00000017 000051d5 e90b8c6c
7a01a980 e913a0c0 fffc44d7 00000003 00000000 00000070 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<78153c96>] __handle_mm_fault+0x29a/0x83b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78153d89>] __handle_mm_fault+0x38d/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S E8CCDE30 0 3726 3722 (NOTLB)
e8ccde44 00200086 00000002 e8ccde30 e8ccde2c 00000000 00000000 ead5e140
00000001 00000000 e9e6cae0 785143e0 0fa415f1 00000017 0001181a e9e6cbec
7a01a980 e93147c0 fffc44d7 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<78153c96>] __handle_mm_fault+0x29a/0x83b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78153d89>] __handle_mm_fault+0x38d/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd S 00000000 0 3727 3722 (NOTLB)
e8cede44 00200086 00000000 00000000 7813e7a9 00000000 00000000 ead5e140
00000001 00000001 f7cb8ae0 e9e6c0b0 0fbbed36 00000017 0003651f f7cb8bec
7a022980 f62ad040 f7cb9010 f7cb8ae0 781299b4 e8cedea0 7fffffff 7fffffff
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<78153c96>] __handle_mm_fault+0x29a/0x83b
[<7840d623>] _spin_unlock+0x25/0x3b
[<78153d89>] __handle_mm_fault+0x38d/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
zabbix_agentd D E8CEFCB0 0 3728 3722 (NOTLB)
e8cefcc4 00200082 00000002 e8cefcb0 e8cefcac 00000000 7840d6e3 7a1d2000
00000008 00000001 e9e6c0b0 f7fbe030 abd7fd2c 0001ff61 00002173 e9e6c1bc
7a022980 e913a340 219905eb 00000003 00000000 00000000 e8cefcd4 2199064e
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7814ce3d>] get_page_from_freelist+0x293/0x35e
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<78153e4d>] __handle_mm_fault+0x451/0x83b
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781197d3>] do_page_fault+0x278/0x535
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<7811955b>] do_page_fault+0x0/0x535
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S E8849EC8 0 3791 1 (NOTLB)
e8849edc 00000086 00000002 e8849ec8 e8849ec4 00000000 00000000 00000000
00000007 00000001 f7fb4b60 f7fbe030 6d0455f6 00000017 0018bcb8 f7fb4c6c
7a022980 f6b15d40 fffc4afe 00000003 00000000 00000000 7fffffff f539f000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3793 1 (NOTLB)
e8d19edc 00000086 00000000 00000000 f53a5c8c ea120a60 00000000 00000000
00000007 00000001 ea120a60 f7e12030 70e57c7e 00000017 0010cf04 ea120b6c
7a022980 e9314040 ea120f98 ea120a60 00000000 00000000 7fffffff f53a5800
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3794 1 (NOTLB)
e8d61edc 00000086 00000000 00000000 f56dcc8c f7e12030 00000000 00000000
00000007 00000001 f7e12030 ec8deb20 70f48c60 00000017 000f0fe2 f7e1213c
7a022980 f62adcc0 f7e12568 f7e12030 00000000 00000000 7fffffff f56dc800
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3795 1 (NOTLB)
e8d23edc 00000086 00000000 00000000 f52fcc8c ec8deb20 00000000 00000000
00000007 00000001 ec8deb20 ec8ff590 71036a29 00000017 000eddc9 ec8dec2c
7a022980 e913a840 ec8df058 ec8deb20 00000000 00000000 7fffffff f52fc800
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3796 1 (NOTLB)
e8903edc 00000086 00000000 00000000 f583748c ec8ff590 00000000 00000000
00000007 00000001 ec8ff590 e8eea130 71118e5d 00000017 000e2434 ec8ff69c
7a022980 f6f5b080 ec8ffac8 ec8ff590 00000000 00000000 7fffffff f5837000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S E8921EC8 0 3797 1 (NOTLB)
e8921edc 00000086 00000002 e8921ec8 e8921ec4 00000000 00000000 00000000
00000007 00000001 e8eea130 f7fbe030 713eaf24 00000017 00023594 e8eea23c
7a022980 e913a5c0 fffc4b41 00000003 00000000 00000000 7fffffff f7242000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3798 1 (NOTLB)
e8923edc 00000086 00000000 00000000 f5d5048c ec906030 00000000 00000000
00000007 00000001 ec906030 ea024070 712da585 00000017 000ebb02 ec90613c
7a022980 e913aac0 ec906568 ec906030 00000000 00000000 7fffffff f5d50000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
agetty S 00000000 0 3799 1 (NOTLB)
e8925edc 00000086 00000000 00000000 f6f2348c ea024070 00000000 00000000
00000007 00000001 ea024070 e8eea130 713c7990 00000017 000ed40b ea02417c
7a022980 f6e31580 ea0245a8 ea024070 00000000 00000000 7fffffff f6f23000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cf28>] read_chan+0x2fb/0x54b
[<78297ed7>] tty_ldisc_try+0x2e/0x32
[<7812027d>] default_wake_function+0x0/0xc
[<7829cc2d>] read_chan+0x0/0x54b
[<7829a215>] tty_read+0x67/0xaf
[<7829a1ae>] tty_read+0x0/0xaf
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S E3429DC4 0 3835 3663 (NOTLB)
e3429dd8 00000082 00000002 e3429dc4 e3429dc0 00000000 00000000 e12799ac
00000007 00000000 7a1eaa60 785143e0 1e0f5b36 00000025 00001291 7a1eab6c
7a01a980 e8d34d00 fffd3160 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S E6C59B4C 0 3838 3835 (NOTLB)
e6c59b60 00000096 00000002 e6c59b4c e6c59b48 00000000 e90b8688 e90b8668
0000000a 00000000 e90b8130 785143e0 28faff08 0001ff23 0001a008 e90b823c
7a01a980 f71bd580 2194e9e6 00000003 00000000 00000000 7fffffff e6c59f9c
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cb0e>] normal_poll+0x0/0x11f
[<78298d67>] tty_poll+0x56/0x63
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7839dedc>] ip_output+0x270/0x2ac
[<7839c7d5>] ip_finish_output+0x0/0x212
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7839b270>] dst_output+0x0/0x7
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<7840d1f4>] _spin_lock_bh+0x38/0x43
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<78370f09>] sock_aio_write+0xcb/0xd7
[<78164133>] do_sync_write+0xc7/0x10a
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash S EA120030 0 3839 3838 (NOTLB)
e3c81f34 00000082 7854ec14 ea120030 00000001 786b7e90 ea120560 ea120560
00000009 00000000 ea120030 e7428030 3ac85a37 00000028 00019fc2 ea12013c
7a01a980 f71bd800 00000000 00000246 7854ec00 00000001 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash S E7428030 0 3866 3839 (NOTLB)
e5f65f34 00000082 7854ec14 e7428030 00000001 786b7e90 e7428560 e7428560
00000008 00000000 e7428030 f7c6c070 8cd13ab9 0001ff22 0001ae9d e742813c
7a01a980 f71bda80 00000000 00000246 7854ec00 00000001 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S 00000127 0 3878 3663 (NOTLB)
e3d2bdd8 00000082 00000000 00000127 00000000 00000000 00000000 e4c4c0ac
00000007 00000001 e21fa0b0 e21fb510 5abe5861 0000002b 00001417 e21fa1bc
7a022980 f71bd080 e21fa5e0 e21fa0b0 00000000 e21fa0b0 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78160ec2>] kmem_cache_free+0x9a/0xa1
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sshd S 00000000 0 3880 3878 (NOTLB)
e406fb60 00000096 00000046 00000000 00000002 00000001 e21fba68 e21fba48
0000000a 00000000 e21fb510 e4632130 adc501bc 0001ff61 00010239 e21fb61c
7a01a980 e93142c0 e91624c0 e1acb80c 7813e99c e91624c0 7fffffff e406ff9c
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840b667>] schedule_timeout+0x13/0x8d
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<7829cb0e>] normal_poll+0x0/0x11f
[<78298d67>] tty_poll+0x56/0x63
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7839dedc>] ip_output+0x270/0x2ac
[<7839c7d5>] ip_finish_output+0x0/0x212
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7839b270>] dst_output+0x0/0x7
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7840d6ef>] _spin_unlock_irqrestore+0x40/0x58
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<7840d1f4>] _spin_lock_bh+0x38/0x43
[<783a3340>] tcp_sendmsg+0x91c/0xa0d
[<78120273>] try_to_wake_up+0x393/0x39d
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<78370f09>] sock_aio_write+0xcb/0xd7
[<78164133>] do_sync_write+0xc7/0x10a
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816f340>] sys_select+0xd6/0x188
[<78103dc6>] sysenter_past_esp+0x8f/0x99
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash S E401D550 0 3881 3880 (NOTLB)
e401ff34 00000082 7854ec14 e401d550 00000001 786b7e90 e401da80 e401da80
00000009 00000000 e401d550 e8eeab60 53749886 0000002c 0001b062 e401d65c
7a01a980 e4743d40 00000000 00000246 7854ec00 00000001 ffffffff 00000001
Call Trace:
[<781276ea>] do_wait+0x8eb/0x9e6
[<7812027d>] default_wake_function+0x0/0xc
[<78127816>] sys_wait4+0x31/0x34
[<78127840>] sys_waitpid+0x27/0x2b
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
bash R running 0 3887 3881 (NOTLB)
unlinkd S 00000218 0 1139 3606 (NOTLB)
c1d17e58 00000086 00000002 00000218 7a0233e0 00000002 00000000 89332b20
00000007 00000001 89332b20 f7cd1490 861dbffc 0001e62a 00045f0d 89332c2c
7a022980 f6e11a40 1ff0a94d 00000003 00000000 00000218 f5a60c00 f7cb5b78
Call Trace:
[<7816943d>] pipe_wait+0x53/0x73
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78169b7d>] pipe_read+0x2af/0x321
[<78174b7c>] file_update_time+0x22/0x6a
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<78164176>] do_sync_read+0x0/0x10a
[<781649a0>] vfs_read+0x88/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
pdflush S ACC9FF9C 0 944 2 (L-TLB)
acc9ffb0 00000092 00000002 acc9ff9c acc9ff98 00000000 9c505590 9c505590
00000007 00000001 7a1fa0b0 f7fbe030 6ce9b6b3 0001e655 0000065b 7a1fa1bc
7a022980 f68e9340 1ff37b50 00000003 00000000 00000000 acc9ffc8 7814e33a
Call Trace:
[<7814e33a>] pdflush+0x0/0x1a6
[<7814e3e8>] pdflush+0xae/0x1a6
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
amavisd S 00000000 0 6000 2823 (NOTLB)
929dfe44 00000086 00000000 00000000 7813e7a9 00000000 00000000 f532f940
0000000a 00000001 e8eeb590 f7e03550 a54ef8a7 0001e62a 000028bb e8eeb69c
7a022980 d3e4a040 e8eebac0 e8eeb590 781299b4 fffffe00 7fffffff 7fffffff
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<781332be>] __rcu_process_callbacks+0xe3/0x172
[<781332a7>] __rcu_process_callbacks+0xcc/0x172
[<7840d623>] _spin_unlock+0x25/0x3b
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squidauth.pl S 95039DC4 0 6285 3606 (NOTLB)
95039dd8 00000082 00000002 95039dc4 95039dc0 00000000 00000000 c31076ac
00000005 00000001 7f498aa0 f7fbe030 84d480c7 0001e4d0 00001b01 7f498bac
7a022980 f6e117c0 1fd9ea3c 00000003 00000000 00000000 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7814aada>] generic_file_aio_write+0x61/0xb6
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squidauth.pl S 00000127 0 6286 3606 (NOTLB)
bc6c7dd8 00000082 00000000 00000127 00000000 00000000 00000000 d18f63ac
00000007 00000001 e7429490 f7e2b550 15274906 0001df92 0004a752 e742959c
7a022980 f68e9d40 e74299c0 e7429490 00000000 e7429490 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7814aada>] generic_file_aio_write+0x61/0xb6
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78165177>] __fput+0x10a/0x134
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
squidauth.pl S 00000127 0 6287 3606 (NOTLB)
e31d5dd8 00000082 00000000 00000127 00000000 00000000 00000000 9cb76cac
00000007 00000001 f7e2b550 7f498aa0 152ad54d 0001df92 00038c47 f7e2b65c
7a022980 c053dac0 f7e2ba80 f7e2b550 00000000 f7e2b550 7fffffff 00000000
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<783ccb1d>] unix_stream_recvmsg+0x1f2/0x475
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78370fdd>] sock_aio_read+0xc8/0xd4
[<7814aada>] generic_file_aio_write+0x61/0xb6
[<7816423d>] do_sync_read+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78165177>] __fput+0x10a/0x134
[<781649b4>] vfs_read+0x9c/0x10a
[<78164da8>] sys_read+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
pdflush S F6A75F9C 0 6288 2 (L-TLB)
f6a75fb0 00000092 00000002 f6a75f9c f6a75f98 00000000 00000292 edcd0560
0000000a 00000001 edcd0030 f7fbe030 bc332535 0001ff60 00002604 edcd013c
7a022980 f6e31800 2198f62b 00000003 00000000 00000000 f6a75fc8 7814e33a
Call Trace:
[<7814e33a>] pdflush+0x0/0x1a6
[<7814e3e8>] pdflush+0xae/0x1a6
[<781350d5>] kthread+0x38/0x5f
[<7813509d>] kthread+0x0/0x5f
[<78104a8f>] kernel_thread_helper+0x7/0x10
=======================
amavisd S 85DD5E30 0 6350 2823 (NOTLB)
85dd5e44 00000086 00000002 85dd5e30 85dd5e2c 00000000 00000000 f532f940
00000007 00000001 a163eaa0 f7fbe030 bcc842a9 0001e62a 000024c3 a163ebac
7a022980 f62ad2c0 1ff0acdf 00000003 00000000 00000000 7fffffff 7fffffff
Call Trace:
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a0b90>] inet_csk_accept+0x9e/0x214
[<78371564>] sock_attach_fd+0x53/0xb5
[<7813518d>] autoremove_wake_function+0x0/0x35
[<783bb524>] inet_accept+0x1f/0xa4
[<78371564>] sock_attach_fd+0x53/0xb5
[<7837262e>] sys_accept+0xb2/0x187
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<783727bc>] sys_socketcall+0xb9/0x242
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103e68>] restore_nocheck+0x12/0x15
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
amavisd S F7E03550 0 6922 2823 (NOTLB)
d7bdfb60 00000096 786b7ed0 f7e03550 00000001 00000046 00000000 00000000
0000000a 00000001 f7e03550 edcd0a60 6a543cc5 0001e655 0008e72a f7e0365c
7a022980 f6e11cc0 7813e99c 00000000 00000046 f5829698 7fffffff d7bdff9c
Call Trace:
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<783b133f>] tcp_v4_rcv+0x3de/0x787
[<7840d623>] _spin_unlock+0x25/0x3b
[<783b1465>] tcp_v4_rcv+0x504/0x787
[<7839949e>] ip_local_deliver+0x1aa/0x1c8
[<78398c10>] ip_local_deliver_finish+0x0/0x14c
[<783992bb>] ip_rcv+0x498/0x4d1
[<7839896c>] ip_rcv_finish+0x0/0x2a4
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c73c>] process_backlog+0xef/0xfa
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7837c9d8>] net_rx_action+0x162/0x18f
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7838108c>] neigh_resolve_output+0x1f2/0x224
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<7812938d>] _local_bh_enable+0xb1/0xbe
[<781197d3>] do_page_fault+0x278/0x535
[<7816f340>] sys_select+0xd6/0x188
[<78103e68>] restore_nocheck+0x12/0x15
[<7811955b>] do_page_fault+0x0/0x535
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
amavisd D 00000000 0 7227 2823 (NOTLB)
d1953e2c 00000082 00000000 00000000 7813e7a9 edcd0a60 7840d6e3 7a1d2000
00000007 00000001 edcd0a60 f7cb14d0 abd73b81 0001ff61 000032a5 edcd0b6c
7a022980 e632d2c0 7a1d2000 7812ccc1 00000000 00000292 d1953e3c 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<78152ded>] do_wp_page+0x3c4/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<78138305>] down_read_trylock+0x47/0x4e
[<7811976d>] do_page_fault+0x212/0x535
[<7811955b>] do_page_fault+0x0/0x535
[<7840d96a>] error_code+0x72/0x78
=======================
amavisd S 00000002 0 7836 2823 (NOTLB)
98947b60 00000096 00000000 00000002 7840ae59 00000000 00000000 00000000
0000000a 00000000 ec8feb60 89333550 bca917ba 0001e62a 00001d50 ec8fec6c
7a01a980 e8d34a80 1ff0ab63 00000003 00000000 00000000 7fffffff 98947f9c
Call Trace:
[<7840ae59>] __sched_text_start+0x701/0x810
[<7840b667>] schedule_timeout+0x13/0x8d
[<783a2023>] tcp_poll+0x1c/0x129
[<7816ec1d>] do_select+0x399/0x3e7
[<7816f1be>] __pollwait+0x0/0xac
[<7812027d>] default_wake_function+0x0/0xc
[<7812027d>] default_wake_function+0x0/0xc
[<7813e986>] trace_hardirqs_on+0x10c/0x14c
[<7813e7a9>] mark_held_locks+0x39/0x53
[<781298c7>] local_bh_enable+0x110/0x130
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7837cbf7>] dev_queue_xmit+0x1f2/0x213
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<781299b4>] local_bh_enable_ip+0xcd/0xed
[<7816ef17>] core_sys_select+0x2ac/0x2c9
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d623>] _spin_unlock+0x25/0x3b
[<78152de2>] do_wp_page+0x3b9/0x406
[<7840d1b1>] _spin_lock+0x33/0x3e
[<781541bf>] __handle_mm_fault+0x7c3/0x83b
[<781197d3>] do_page_fault+0x278/0x535
[<7816f340>] sys_select+0xd6/0x188
[<78103e68>] restore_nocheck+0x12/0x15
[<7811955b>] do_page_fault+0x0/0x535
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
sadc D 00000000 0 8202 1 (NOTLB)
ccff9cc4 00000082 00000000 00000000 7813e7a9 edcd1490 7840d6e3 7a1d2000
00000007 00000001 edcd1490 e9e6c0b0 abd7dbb9 0001ff61 0000216c edcd159c
7a022980 c053d340 7a1d2000 7812ccc1 00000000 00000296 ccff9cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<7814cb18>] free_hot_cold_page+0x13b/0x167
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781648dd>] vfs_write+0xd1/0x10c
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
emerge D B7BF9CB0 0 8204 3061 (NOTLB)
b7bf9cc4 00200082 00000002 b7bf9cb0 b7bf9cac 00000000 7840d6e3 785cde80
00000007 00000000 89333550 785143e0 abd783fe 0001ff61 00002d5e 8933365c
7a01a980 c053d840 219905ed 00000003 00000000 00000000 b7bf9cd4 2199064e
Call Trace:
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<7813f8ca>] __lock_acquire+0xaf0/0xb84
[<7814ce3d>] get_page_from_freelist+0x293/0x35e
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<78153e4d>] __handle_mm_fault+0x451/0x83b
[<7813518d>] autoremove_wake_function+0x0/0x35
[<781197d3>] do_page_fault+0x278/0x535
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<7811955b>] do_page_fault+0x0/0x535
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
mkpir D 00000000 0 8297 3061 (NOTLB)
9c507cc4 00000082 00000000 00000000 7813e7a9 9c505590 7840d6e3 7a1d2000
00000007 00000001 9c505590 f7cb9510 abd79872 0001ff61 00002bf0 9c50569c
7a022980 f68e9340 7a1d2000 7812ccc1 00000000 00000296 9c507cd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<781a8658>] journal_stop+0x1ce/0x1da
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78148be7>] file_read_actor+0x0/0xde
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<78165177>] __fput+0x10a/0x134
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
screen S 8DBAFF98 0 8364 3866 (NOTLB)
8dbaffac 00000086 00000002 8dbaff98 8dbaff94 00000000 00000000 7812850a
00000007 00000000 f7c6c070 785143e0 455dd02b 0001ff61 00005ab9 f7c6c17c
7a01a980 e913ad40 2198ff2e 00000003 00000000 00000000 ffffffff 00000000
Call Trace:
[<7812850a>] do_setitimer+0x16e/0x30b
[<7812f315>] sys_pause+0x11/0x17
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
screen D 00000000 0 8365 8364 (NOTLB)
eca0fcc4 00000082 00000000 00000000 7813e7a9 f7cb14d0 7840d6e3 7a1d2000
00000007 00000001 f7cb14d0 9c505590 abd76c82 0001ff61 00003101 f7cb15dc
7a022980 d3e4acc0 7a1d2000 7812ccc1 00000000 00000296 eca0fcd4 2199064e
Call Trace:
[<7813e7a9>] mark_held_locks+0x39/0x53
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812ccc1>] __mod_timer+0x92/0x9c
[<7840b6c4>] schedule_timeout+0x70/0x8d
[<7840d6e3>] _spin_unlock_irqrestore+0x34/0x58
[<7812cada>] process_timeout+0x0/0x5
[<7840b0ce>] io_schedule_timeout+0x1e/0x28
[<78151eca>] congestion_wait+0x50/0x64
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7814dd9f>] balance_dirty_pages_ratelimited_nr+0x16e/0x1dc
[<7814a483>] generic_file_buffered_write+0x4ee/0x605
[<7814a8cf>] __generic_file_aio_write_nolock+0x335/0x4df
[<78128f60>] current_fs_time+0x41/0x46
[<7814aa1a>] __generic_file_aio_write_nolock+0x480/0x4df
[<7813e99c>] trace_hardirqs_on+0x122/0x14c
[<7814aad1>] generic_file_aio_write+0x58/0xb6
[<78199bb4>] ext3_file_write+0x24/0x92
[<78164133>] do_sync_write+0xc7/0x10a
[<7813518d>] autoremove_wake_function+0x0/0x35
[<7840c019>] __mutex_lock_slowpath+0x25f/0x267
[<7816406c>] do_sync_write+0x0/0x10a
[<78164896>] vfs_write+0x8a/0x10c
[<78164e0f>] sys_write+0x41/0x67
[<78103d96>] sysenter_past_esp+0x5f/0x99
=======================
Are any IO errors occurring at all?
No.
Best regards,
Krzysztof Olędzki