Re: Process hang with 2.6.6-rc1

[Shantanu Goel]

Apologies if you see this message truncated earlier.
Attached is the complete report.
-------------------------------------------------------
Hi,

I have been experiencing process hangs since upgrading
to 2.6.6-rc1.  The system is usually able to stay up
for a day before the hang.  But it is quite consistent
in that it always hangs.  I experienced the same hang
with 2.6.5-bk1 as well.  In the thread traces, it
looks as though the mmap_sem of pid 3960 (pan) is not
being released preventing any other process from
locking the address space.  This is readily apparent
from the hung "ps" command.  It is trying to read
"/proc/3960/stat".
The machine is otherwise usable but trying to shut it
down via reboot hangs it.

Any clues would be appreciated as I have not been able
to make any headway in debugging this.  My hardware
information is below and I have attached the sysrq-t
output and .config.

Thanks,
Shantanu

Kernel version: 2.6.6-rc1 SMP

Hardware Info: Dell PE1600SC 2x2Ghz Xeon CPUs
               1 18GB SCSI disk (root)
               1 250GB Maxtor external USB 2.0 disk

Last few lines from "ps" trace:

open("/proc/3915/cmdline", O_RDONLY)    = 6
read(6, "/usr/lib/mozilla-1.4.1/mozilla-b"..., 2047) =
51
close(6)                                = 0
open("/proc/3915/environ", O_RDONLY)    = 6
read(6, "SSH_AGENT_PID=3230\0HOSTNAME=anu."..., 2047)
= 1134
close(6)                                = 0
stat64("/proc/3960", {st_mode=S_IFDIR|0555, st_size=0,
...}) = 0
open("/proc/3960/stat", O_RDONLY)       = 6
read(6, 


	
		
__________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online by April 15th
http://taxes.yahoo.com/filing.htmlSysRq : Show State

                                               sibling
  task             PC      pid father child younger older
init          S 00000001  4708     1      0     2               (NOTLB)
c1a7de90 00000082 c1a21ce0 00000001 0000000f c11ff880 00000000 c1a22660 
       c1a7de90 00000246 c1a22660 c1a7dea4 c1a7deb0 c1a21ce0 00000000 053ab0c0 
       000f9465 f7f916b0 f7f91868 f7f916b0 00000010 00000000 05623bf3 c1a7dea4 
Call Trace:
 [<c02cc2f1>] schedule_timeout+0x6e/0xbf
 [<c0126f65>] process_timeout+0x0/0xc
 [<c0173438>] do_select+0x183/0x2d4
 [<c0173127>] __pollwait+0x0/0xac
 [<c017384d>] sys_select+0x299/0x4cc
 [<c0105e25>] sysenter_past_esp+0x52/0x71

migration/0   S F721BFC0  7868     2      1             3       (L-TLB)
f7f77f9c 00000046 00000001 f721bfc0 00000001 f7f77f70 c011944f f72a4c70 
       00000003 00000000 f721bfc0 f721bfb4 f7292750 c1a11ce0 00000000 4f444540 
       000f41ff f7f90030 f7f901e8 00000000 f7f916b0 c1a11ce0 c1a1262c c1a11ce0 
Call Trace:
 [<c011944f>] __wake_up_common+0x38/0x57
 [<c011aa9a>] migration_thread+0x94/0x15c
 [<c013202d>] kthread_exit_files+0x33/0x53
 [<c0132104>] kthread+0xb7/0xbc
 [<c011aa06>] migration_thread+0x0/0x15c
 [<c013204d>] kthread+0x0/0xbc
 [<c0104021>] kernel_thread_helper+0x5/0xb

ksoftirqd/0   S 00000001  6168     3      1             4     2 (L-TLB)
f7f7ffa0 00000046 c1a11ce0 00000001 0000000f 00c8b320 c035947c 00000000 
       f7f7e000 f7f7ff7c c012299e 00000000 c03cb6dc c1a11ce0 00000000 748cb240 
       000f941d f7f75730 f7f758e8 00000246 f7f7e000 ffffe000 f7f7e000 f7f7e000 
Call Trace:
 [<c012299e>] tasklet_action+0x68/0xb1
 [<c0122bd0>] ksoftirqd+0xaa/0xc5
 [<c0132104>] kthread+0xb7/0xbc
 [<c0122b26>] ksoftirqd+0x0/0xc5
 [<c013204d>] kthread+0x0/0xbc
 [<c0104021>] kernel_thread_helper+0x5/0xb

migration/1   S F7219FC0  7868     4      1             5     3 (L-TLB)
f7f7df9c 00000046 00000001 f7219fc0 00000001 f7f7df70 c011944f f76b1190 
       00000003 00000000 f7219fc0 f7219fb4 f76ca330 c1a19ce0 00000000 2b71bd00 
       000f41ff f7f75190 f7f75348 00000000 f7f75730 c1a11ce0 c1a1a62c c1a19ce0 
Call Trace:
 [<c011944f>] __wake_up_common+0x38/0x57
 [<c011aa9a>] migration_thread+0x94/0x15c
 [<c013202d>] kthread_exit_files+0x33/0x53
 [<c0132104>] kthread+0xb7/0xbc
 [<c011aa06>] migration_thread+0x0/0x15c
 [<c013204d>] kthread+0x0/0xbc
 [<c0104021>] kernel_thread_helper+0x5/0xb

ksoftirqd/1   S F7F7BF44  6852     5      1             6     4 (L-TLB)
f7f7bfa0 00000046 f7f7bf44 f7f7bf44 0000000f f6f95764 c1a1b6fc 00000000 
       f7f7a000 f7f7bf7c c012299e 00000000 f76cb410 c1a19ce0 00000000 3c7b5200 
       000f93ad f7f74bf0 f7f74da8 00000246 f7f7a000 ffffe000 f7f7a000 f7f7a000 
Call Trace:
 [<c012299e>] tasklet_action+0x68/0xb1
 [<c0122bd0>] ksoftirqd+0xaa/0xc5
 [<c0132104>] kthread+0xb7/0xbc
 [<c0122b26>] ksoftirqd+0x0/0xc5
 [<c013204d>] kthread+0x0/0xbc
 [<c0104021>] kernel_thread_helper+0x5/0xb

migration/2   S F559BFC0  7868     6      1             7     5 (L-TLB)
f7f79f9c 00000046 00000001 f559bfc0 00000001 f7f79f70 c011944f f76cb410 
       00000003 00000000 f559bfc0 f559bfb4 f4f5e6d0 c1a21ce0 00000000 cce739c0 
       000f4237 f7f74650 f7f74808 00000000 f7f916b0 c1a11ce0 c1a2262c c1a21ce0 
Call Trace:
 [<c011944f>] __wake_up_common+0x38/0x57
 [<c011aa9a>] migration_thread+0x94/0x15c
 [<c013202d>] kthread_exit_files+0x33/0x53
 [<c0132104>] ozilla-bin   S 00000001  5732  3948   3915          3949       (NOTLB)
f2f83ef8 00200086 c1a19ce0 00000001 0000000f c0140606 c0324680 00000000 
       000000d0 f2f11a50 00000001 f2ee9730 00000010 c1a19ce0 00000000 f2cfb980 
       000f9464 f2ee9730 f2ee98e8 f2f83ef0 c0140897 00200246 00000000 7fffffff 
Call Trace:
 [<c0140606>] __alloc_pages+0x97/0x2f3
 [<c0140897>] __get_free_pages+0x35/0x40
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c016cb18>] pipe_poll+0x33/0x79
 [<c0173ad7>] do_pollfd+0x57/0x98
 [<c0173bb2>] do_poll+0x9a/0xb9
 [<c0173d12>] sys_poll+0x141/0x1fc
 [<c015ea04>] sys_read+0x5b/0x5d
 [<c0173127>] __pollwait+0x0/0xac
 [<c0105e25>] sysenter_past_esp+0x52/0x71

mozilla-bin   S 00000001  6744  3949   3915          3950  3948 (NOTLB)
f2ee7e84 00200086 c1a11ce0 00000001 0000000f f2f0c800 00000001 f2d81c00 
       f2ee7e70 00000040 00002260 f4eb4370 f76b00b0 c1a11ce0 00000000 ee806740 
       000f424c f2ee8bf0 f2ee8da8 800f8000 f2ee7e8c c0150fe3 fffffff5 7fffffff 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0132443>] get_futex_key+0x48/0x1dc
 [<c0132a8e>] queue_me+0x4f/0xcb
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c016ca61>] pipe_write+0x33/0x37
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c015ea61>] sys_write+0x5b/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

mozilla-bin   S F2CFB980  6324  3950   3915          6437  3949 (NOTLB)
f24d9e84 00200086 f5e406d0 f2cfb980 000f9464 c03d1cc0 00000000 c1a22660 
       f24d9e84 00200246 f2cfb980 000f9464 f5e406d0 c1a21ce0 00000000 f2cfb980 
       000f9464 f4e505d0 f4e50788 8014e000 f24d9e8c c0150fe3 056238ab f24d9e98 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc2f1>] schedule_timeout+0x6e/0xbf
 [<c0126f65>] process_timeout+0x0/0xc
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c016ca61>] pipe_write+0x33/0x37
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c0105e25>] sysenter_past_esp+0x52/0x71

mozilla-bin   S 00000001  5480  6437   3915                3950 (NOTLB)
ef7d7e84 00200086 c1a29ce0 00000001 0000000f c33b7ee8 ef7d7efc ef7d7e60 
       c013cb1c 0000006f f7ee77f0 c013ce5a 00000000 c1a29ce0 00000000 a1996340 
       000f926b e9de82b0 e9de8468 80103000 ef7d7e8c c0150fe3 fffffff5 7fffffff 
Call Trace:
 [<c013cb1c>] file_read_actor+0x0/0xf2
 [<c013ce5a>] generic_file_aio_read+0x52/0x69
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0132443>] get_futex_key+0x48/0x1dc
 [<c0132b7c>] unqueue_me+0x72/0x10d
 [<c0132a8e>] queue_me+0x4f/0xcb
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01326bb>] futex_wake+0x58/0x126
 [<c0119405>] default_wake_function+0x0/0x12
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c015de72>] sys_close+0x80/0xed
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           S 00000001  5080  3960      1  3963    4150  3915 (NOTLB)
f21b5ef8 00000086 c1a19ce0 00000001 0000000f c0140606 00000000 c1a1a660 
       f21b5ef8 00000246 c1a1a660 f21b5f0c f21b5f18 c1a19ce0 00000000 eb99e900 
       000f9465 f2ee9190 f2ee9348 f21b5ef0 f4eb4058 f2f60c64 0562384a f21b5f0c 
Call Trace:
 [<c0140606>] __alloc_pages+0x97/0x2f3
 [<c02cc2f1>] schedule_timeout+0x6e/0xbf
 [<c0126f65>] process_timeout+0x0/0xc
 [<c0173bb2>] do_poll+0x9a/0xb9
 [<c0173d12>] sys_poll+0x141/0x1fc
 [<c0173127>] __pollwait+0x0/0xac
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           S 758806C0  5224  3963   3960          3964       (NOTLB)
f2efbe84 00000086 f76b00b0 758806c0 000f74a3 c35c3510 f2efbefc 00000246 
       00000000 c1a11ce0 758806c0 000f74a3 f76b00b0 c1a19ce0 00000000 758806c0 
       000f74a3 f2ee8650 f2ee8808 081b0000 f2efbe8c c0150fe3 fffffff5 7fffffff 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0132443>] get_futex_key+0x48/0x1dc
 [<c0132a8e>] queue_me+0x4f/0xcb
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c013275a>] futex_wake+0xf7/0x126
 [<c0119405>] default_wake_function+0x0/0x12
 [<c02cb767>] schedule+0x377/0x71e
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           S C16B3120  5100  3964   3960          3965  3963 (NOTLB)
f2ee5e84 00000086 00000001 c16b3120 000196d3 c0324888 00000010 c1020000 
       c0324888 00000096 ffffffff 00000247 f7f740b0 c1a21ce0 00000000 ab1d2dc0 
       000f74a2 f21b3830 f21b39e8 081b0000 f2ee5e8c c0150fe3 fffffff5 7fffffff 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0132443>] get_futex_key+0x48/0x1dc
 [<c0132a8e>] queue_me+0x4f/0xcb
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c013275a>] futex_wake+0xf7/0x126
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0151163>] unmap_vma_list+0x1d/0x2a
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           S 00000001  5176  3965   3960          3966  3964 (NOTLB)
f24ebe84 00000086 c1a21ce0 00000001 0000000f c0324888 00000010 c1020000 
       c0324888 00000096 ffffffff 00000247 d71fe1f4 c1a21ce0 000186a0 3fb55700 
       000f74a3 f5f4d190 f5f4d348 081b0000 f24ebe8c c0150fe3 fffffff5 7fffffff 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0132443>] get_futex_key+0x48/0x1dc
 [<c0132a8e>] queue_me+0x4f/0xcb
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c013275a>] futex_wake+0xf7/0x126
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0151163>] unmap_vma_list+0x1d/0x2a
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           D 00000001  5240  3966   3960          3967  3965 (NOTLB)
f24edd68 00000086 c1a11ce0 00000001 0000000f 00000001 c039c388 0000000a 
       f24edd3c c01226f8 c039c388 00000046 00000000 c1a11ce0 000186a0 90884200 
       000f74a3 f6f08b70 f6f08d28 00000000 80000800 f7fffe80 f6fa0b98 f6f08b70 
Call Trace:
 [<c01226f8>] __do_softirq+0xac/0xae
 [<c01c71e9>] rwsem_down_failed_common+0xa4/0x17e
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c02cc3f8>] rwsem_down_read_failed+0x29/0x32
 [<c011680d>] .text.lock.fault+0x1b/0x82
 [<c02779bb>] release_sock+0x6c/0xcb
 [<c0277909>] lock_sock+0x6f/0xb5
 [<c029a6c9>] tcp_recvmsg+0x2e9/0x701
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c0106979>] error_code+0x2d/0x38
 [<c015007b>] vma_link+0x8/0xfd
 [<c0152a1c>] move_one_page+0x32/0x1a1
 [<c0152bc5>] move_page_tables+0x3a/0x4b
 [<c0152c64>] move_vma+0x8e/0x1b4
 [<c015302a>] do_mremap+0x2a0/0x3e8
 [<c01531da>] sys_mremap+0x68/0x8e
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           S F74BB118  5080  3967   3960          3968  3966 (NOTLB)
f4d95e84 00000086 00000001 f74bb118 00003dd3 c0324888 00000010 c1020000 
       c0324888 00000096 ffffffff 00000247 f7f740b0 c1a21ce0 000186a0 0b411b40 
       000f74a2 f21b3290 f21b3448 081b0000 f4d95e8c c0150fe3 fffffff5 7fffffff 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0132443>] get_futex_key+0x48/0x1dc
 [<c0132a8e>] queue_me+0x4f/0xcb
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c013275a>] futex_wake+0xf7/0x126
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0118ed6>] scheduler_tick+0x10d/0x63c
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c01132e0>] smp_apic_timer_interrupt+0xd9/0x141
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pan           D 00000001  5004  3968   3960                3967 (NOTLB)
f2ee3ea8 00000086 c1a29ce0 00000001 0000000f f2ee3e70 c01226f8 c039c388 
       54f7d008 fffc3df4 eef4754c f2ee3ec8 c014eb7b c1a29ce0 000f4240 90978440 
       000f74a3 f21b2750 f21b2908 54f73004 f2ee3f04 f6fa0b78 f6fa0b98 f21b2750 
Call Trace:
 [<c01226f8>] __do_softirq+0xac/0xae
 [<c014eb7b>] do_no_page+0x5d/0x3d5
 [<c01c71e9>] rwsem_down_failed_common+0xa4/0x17e
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c02cc3f8>] rwsem_down_read_failed+0x29/0x32
 [<c011680d>] .text.lock.fault+0x1b/0x82
 [<c0126f65>] process_timeout+0x0/0xc
 [<c0117a5e>] recalc_task_prio+0x90/0x1aa
 [<c0118ed6>] scheduler_tick+0x10d/0x63c
 [<c01226f8>] __do_softirq+0xac/0xae
 [<c01078ae>] do_nmi+0x56/0x58
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c0106979>] error_code+0x2d/0x38

bash          S 00000001  5464  3988   3331  6766    4025  3333 (NOTLB)
cb533f34 00200086 c1a21ce0 00000001 0000000f 3f5fe065 cba290ec f65c02ac 
       00000001 00000001 f65c0278 f65c0298 cb526428 c1a21ce0 00000000 f45a4980 
       000f9295 f21b2cf0 f21b2ea8 00000001 cb532000 cb533f3c fffffe00 f21b2cf0 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 00000001  5420  4025   3331          4073  3988 (NOTLB)
dcb53e30 00200082 c1a19ce0 00000001 0000000f f4ea26f8 f21b21b0 c1a19ce0 
       f7f65fb4 00000016 f5f15000 00000016 dcb53e2c c1a19ce0 00000000 e9642ac0 
       000f9298 f21b21b0 f21b2368 dcb53e48 00000016 00000016 f10a3000 7fffffff 
Call Trace:
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c014ddd4>] do_wp_page+0x8d/0x3e5
 [<c01fdf41>] write_chan+0x153/0x219
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c012a29b>] sys_rt_sigprocmask+0x97/0x140
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 195AEFC0  5416  4073   3331  6196    4248  4025 (NOTLB)
c5ff9f34 00200086 f5ff85d0 195aefc0 000f7250 0438f065 dc8cd430 f4ea24ec 
       00000001 00000001 195aefc0 000f7250 f5ff85d0 c1a11ce0 00000000 195aefc0 
       000f7250 c62ab830 c62ab9e8 00000001 c5ff8000 c5ff9f3c fffffe00 c62ab830 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

run-mozilla.s S 00000001  5048  4150      1  4164    4170  3960 (NOTLB)
c3e2df34 00200086 c1a19ce0 00000001 0000000f 4b989065 c3d69fc4 c7c2a4ec 
       c3e2df10 c012eff8 c7c2a4b8 c7c2a4d8 cb5262ac c1a19ce0 00000000 57b49680 
       000f428b c94728d0 c9472a88 00000001 080e0d88 c94728d0 fffffe00 c94728d0 
Call Trace:
 [<c012eff8>] attach_pid+0x21/0xd2
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

firefox-bin   S 00000001  4620  4164   4150  4168               (NOTLB)
c3d69ef8 00200086 c1a21ce0 00000001 0000000f 00000001 c9472330 00000010 
       00000000 000000d0 00000000 000000d0 f76b00b0 c1a21ce0 00000000 e8807400 
       000f9465 c9472330 c94724e8 c017319d f5c49a68 00200246 00000000 7fffffff 
Call Trace:
 [<c017319d>] __pollwait+0x76/0xac
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0274a64>] sock_poll+0x29/0x30
 [<c0173ad7>] do_pollfd+0x57/0x98
 [<c0173bb2>] do_poll+0x9a/0xb9
 [<c0173d12>] sys_poll+0x141/0x1fc
 [<c0173127>] __pollwait+0x0/0xac
 [<c0105e25>] sysenter_past_esp+0x52/0x71

firefox-bin   S 00000001  5724  4168   4164          4171       (NOTLB)
c4d27ef8 00200086 c1a21ce0 00000001 0000000f c0140606 c0324680 00000000 
       000000d0 c01078ae 00000001 cf2376b0 00000010 c1a21ce0 00000000 843e2c00 
       000f9463 cf2376b0 cf237868 c4d27ef0 c0140897 00200246 00000000 7fffffff 
Call Trace:
 [<c0140606>] __alloc_pages+0x97/0x2f3
 [<c01078ae>] do_nmi+0x56/0x58
 [<c0140897>] __get_free_pages+0x35/0x40
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c016cb18>] pipe_poll+0x33/0x79
 [<c0173ad7>] do_pollfd+0x57/0x98
 [<c0173bb2>] do_poll+0x9a/0xb9
 [<c0173d12>] sys_poll+0x141/0x1fc
 [<c015ea04>] sys_read+0x5b/0x5d
 [<c0173127>] __pollwait+0x0/0xac
 [<c0105e25>] sysenter_past_esp+0x52/0x71

firefox-bin   S 00000001  5936  4171   4164                4168 (NOTLB)
c772de84 00200086 c1a19ce0 00000001 0000000f c772de98 00000000 c1a1a660 
       c772de84 00200246 c1a1a660 c772de98 f76b00b0 c1a19ce0 00000000 e87131c0 
       000f9465 f6fd68d0 f6fd6a88 08aee000 c772de8c c0150fe3 05623885 c772de98 
Call Trace:
 [<c0150fe3>] find_extend_vma+0x27/0x79
 [<c02cc2f1>] schedule_timeout+0x6e/0xbf
 [<c0126f65>] process_timeout+0x0/0xc
 [<c0132d44>] futex_wait+0x12d/0x193
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c016ca61>] pipe_write+0x33/0x37
 [<c013307f>] do_futex+0x6d/0x7d
 [<c013316d>] sys_futex+0xde/0xea
 [<c0105e25>] sysenter_past_esp+0x52/0x71

gconfd-2      S 00000001  5364  4170      1                4150 (NOTLB)
c70a9ef8 00200086 c1a11ce0 00000001 0000000f c0140606 00000000 c1a12660 
       c70a9ef8 00200246 c1a12660 c70a9f0c c70a9f18 c1a11ce0 00000000 8591a580 
       000f944c f66fd8b0 f66fda68 c70a9ef0 f5c49c00 c62b5730 056261c5 c70a9f0c 
Call Trace:
 [<c0140606>] __alloc_pages+0x97/0x2f3
 [<c02cc2f1>] schedule_timeout+0x6e/0xbf
 [<c0126f65>] process_timeout+0x0/0xc
 [<c0173bb2>] do_poll+0x9a/0xb9
 [<c0173d12>] sys_poll+0x141/0x1fc
 [<c0173127>] __pollwait+0x0/0xac
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 00000001  5440  4248   3331          4364  4073 (NOTLB)
ca437e30 00200086 c1a21ce0 00000001 0000000f f6ee54b8 f66fd310 c1a21ce0 
       f7f73fb4 00000013 c581a000 00000013 ca437e2c c1a21ce0 00000000 65b11880 
       000f924a f66fd310 f66fd4c8 ca437e48 00000013 00000013 cde37000 7fffffff 
Call Trace:
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c014ddd4>] do_wp_page+0x8d/0x3e5
 [<c01fdf41>] write_chan+0x153/0x219
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c012a29b>] sys_rt_sigprocmask+0x97/0x140
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 376C7680  5364  4364   3331  5002    4403  4248 (NOTLB)
cb55bf34 00200082 c94739b0 376c7680 000f50ce 0cb3c065 dc8cddb0 f71ed96c 
       00000001 00000001 376c7680 000f50ce c94739b0 c1a19ce0 00000000 376c7680 
       000f50ce f66fc7d0 f66fc988 00000001 cb55a000 cb55bf3c fffffe00 f66fc7d0 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 00000001  5308  4403   3331          6236  4364 (NOTLB)
c6c9fe30 00200082 c1a19ce0 00000001 0000000f 00000000 000004d2 00000000 
       00000002 00000013 f1081000 00000013 c6c9fe2c c1a19ce0 0001b207 4de56a40 
       000f725f f2ee80b0 f2ee8268 c6c9fe48 00000013 00000013 f0dda000 7fffffff 
Call Trace:
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c020245d>] con_get_trans_new+0x1c/0x56
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c014ddd4>] do_wp_page+0x8d/0x3e5
 [<c01fdf41>] write_chan+0x153/0x219
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c012a29b>] sys_rt_sigprocmask+0x97/0x140
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

rlogin        S 00000001  6204  5002   4364  5003               (NOTLB)
c3b73e30 00200086 c1a21ce0 00000001 0000000f f6c423c0 c3b73e68 c0299234 
       f6c421b8 00000000 00000001 00000000 00000000 c1a21ce0 00000000 f4c9a2c0 
       000f50ce c94739b0 c9473b68 00000000 efae17e0 f75cb92c f589f000 7fffffff 
Call Trace:
 [<c0299234>] tcp_sendmsg+0x479/0x1127
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c02bb201>] inet_sendmsg+0x4b/0x56
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c015e872>] do_sync_write+0x89/0xb4
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c010685c>] common_interrupt+0x18/0x20
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

rlogin        S 00000001  6016  5003   5002                     (NOTLB)
f3283d50 00200086 c1a19ce0 00000001 0000000f c01fbc40 f75cb000 0000001d 
       00020001 f6c423c0 f75cbc14 00010800 00200202 c1a19ce0 00000000 4e7bbe00 
       000f9465 c9472e70 c9473028 0000001d 00020001 f752bafc f6c421b8 7fffffff 
Call Trace:
 [<c01fbc40>] n_tty_receive_buf+0x1b2/0x1670
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0277d19>] kfree_skbmem+0x25/0x2c
 [<c0277d93>] __kfree_skb+0x73/0xdf
 [<c02779bb>] release_sock+0x6c/0xcb
 [<c02a1df8>] tcp_rcv_established+0x141/0x74f
 [<c029a1a7>] tcp_data_wait+0xab/0xaf
 [<c011b4c4>] autoremove_wake_function+0x0/0x4b
 [<c011b4c4>] autoremove_wake_function+0x0/0x4b
 [<c029a753>] tcp_recvmsg+0x373/0x701
 [<c02bb19e>] inet_recvmsg+0x54/0x6c
 [<c027437b>] sock_aio_read+0xbe/0xd1
 [<c015e6b2>] do_sync_read+0x89/0xb4
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01fddee>] write_chan+0x0/0x219
 [<c02cb767>] schedule+0x377/0x71e
 [<c015e7b5>] vfs_read+0xd8/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

rlogin        S EBC6FDEB  6244  6196   4073  6197               (NOTLB)
e844fe30 00200082 f6edec70 ebc6fdeb 000f7250 f679c240 e844fe68 c0299234 
       f679c038 00000000 ebc6fdeb 000f7250 f6edec70 c1a19ce0 00000000 ebd1b880 
       000f7250 f5ff85d0 f5ff8788 00000000 dcc84fd4 f7c6d92c f5b17000 7fffffff 
Call Trace:
 [<c0299234>] tcp_sendmsg+0x479/0x1127
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c02bb201>] inet_sendmsg+0x4b/0x56
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c015e872>] do_sync_write+0x89/0xb4
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

rlogin        S 00000001  6296  6197   6196                     (NOTLB)
cc53dd50 00200082 c1a19ce0 00000001 0000000f c01fbc40 f7c6d000 0000001d 
       00020001 f679c240 f7c6dc14 00010800 00200202 c1a19ce0 00000000 9d666100 
       000f9465 cf236030 cf2361e8 00000100 c0277d19 f720855c f679c038 7fffffff 
Call Trace:
 [<c01fbc40>] n_tty_receive_buf+0x1b2/0x1670
 [<c0277d19>] kfree_skbmem+0x25/0x2c
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0277d19>] kfree_skbmem+0x25/0x2c
 [<c0277d93>] __kfree_skb+0x73/0xdf
 [<c02779bb>] release_sock+0x6c/0xcb
 [<c02a1df8>] tcp_rcv_established+0x141/0x74f
 [<c029a1a7>] tcp_data_wait+0xab/0xaf
 [<c011b4c4>] autoremove_wake_function+0x0/0x4b
 [<c011b4c4>] autoremove_wake_function+0x0/0x4b
 [<c029a753>] tcp_recvmsg+0x373/0x701
 [<c02bb19e>] inet_recvmsg+0x54/0x6c
 [<c027437b>] sock_aio_read+0xbe/0xd1
 [<c015e6b2>] do_sync_read+0x89/0xb4
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01fddee>] write_chan+0x0/0x219
 [<c02cb767>] schedule+0x377/0x71e
 [<c015e7b5>] vfs_read+0xd8/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 00000001  5436  6236   3331                4403 (NOTLB)
d41cfe30 00200086 c1a21ce0 00000001 0000000f 0000000f c0325210 c0324e80 
       d41cfe18 00000011 f08ca000 00000011 d41cfe2c c1a21ce0 0001e848 b9c33c00 
       000f7369 f72921b0 f7292368 d41cfe48 00000011 00000011 d6c9b000 7fffffff 
Call Trace:
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c014ddd4>] do_wp_page+0x8d/0x3e5
 [<c01fdf41>] write_chan+0x153/0x219
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c012a29b>] sys_rt_sigprocmask+0x97/0x140
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

pdflush       S 9D389A40  4972  6344     13                     (L-TLB)
dbddbf5c 00000046 f5d6acf0 9d389a40 000f9465 00000000 00000000 00000000 
       00000000 00000000 9d389a40 000f9465 f5d6acf0 c1a21ce0 00000000 9d389a40 
       000f9465 c9473410 c94735c8 00000000 00000000 00000000 dbdda000 dbddbfa8 
Call Trace:
 [<c0142780>] __pdflush+0xbc/0x312
 [<c0142a00>] pdflush+0x2a/0x2e
 [<c0132104>] kthread+0xb7/0xbc
 [<c01429d6>] pdflush+0x0/0x2e
 [<c013204d>] kthread+0x0/0xbc
 [<c0104021>] kernel_thread_helper+0x5/0xb

pdflush       S 00000001  5452  6355     11                     (L-TLB)
c54c7f5c 00000046 c1a21ce0 00000001 0000000f 00000000 00000000 00000000 
       00000000 00000000 00000000 00000000 00000000 c1a21ce0 00000000 d9730b00 
       000f9436 e9de9930 e9de9ae8 00000000 00000000 00000000 c54c6000 c54c7fa8 
Call Trace:
 [<c0142780>] __pdflush+0xbc/0x312
 [<c0142a00>] pdflush+0x2a/0x2e
 [<c0132104>] kthread+0xb7/0xbc
 [<c01429d6>] pdflush+0x0/0x2e
 [<c013204d>] kthread+0x0/0xbc
 [<c0104021>] kernel_thread_helper+0x5/0xb

ps            D 00000001  5608  6766   3988                     (NOTLB)
f4bc7db8 00200082 c1a21ce0 00000001 0000000f c1a236fc f7fefee0 00000000 
       0000016f 00200246 00000000 00000170 c018d9aa c1a21ce0 005b8d80 f78300c0 
       000f9295 eae98650 eae98808 00000001 00000000 f7c30644 f6fa0b98 eae98650 
Call Trace:
 [<c018d9aa>] proc_alloc_inode+0x1e/0x73
 [<c01c71e9>] rwsem_down_failed_common+0xa4/0x17e
 [<c017a380>] alloc_inode+0x1c/0x141
 [<c02cc3f8>] rwsem_down_read_failed+0x29/0x32
 [<c0192bde>] .text.lock.array+0x5b/0xcd
 [<c018f938>] pid_revalidate+0x23/0x112
 [<c018f951>] pid_revalidate+0x3c/0x112
 [<c01777f7>] dput+0x24/0x2ca
 [<c016e3c0>] link_path_walk+0x713/0xb1c
 [<c014042b>] buffered_rmqueue+0xf5/0x239
 [<c018ec72>] proc_info_read+0x53/0x10e
 [<c015d7de>] filp_open+0x5d/0x5f
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S ABA88D80  6440  6770   3114  6837               (NOTLB)
c221df34 00000086 f76b5110 aba88d80 000f92a1 44d9f065 cba29988 f70064ec 
       00000001 00000001 aba88d80 000f92a1 f76b5110 c1a19ce0 00000000 aba88d80 
       000f92a1 e9de8850 e9de8a08 00000001 c221c000 c221df3c fffffe00 e9de8850 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

ps            D 00000001  6440  6837   6770                     (NOTLB)
c3f8bdb8 00000086 c1a19ce0 00000001 0000000f c03ca104 00000000 00000002 
       c01162b0 c3f8bdd8 c0106979 c3f8bd88 00000002 c1a19ce0 004c4b40 ad16c380 
       000f92a1 f76b5110 f76b52c8 c8763132 00000000 c3f8bde8 f6fa0b98 f76b5110 
Call Trace:
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c0106979>] error_code+0x2d/0x38
 [<c01c71e9>] rwsem_down_failed_common+0xa4/0x17e
 [<c02cc3f8>] rwsem_down_read_failed+0x29/0x32
 [<c0192bde>] .text.lock.array+0x5b/0xcd
 [<c018f938>] pid_revalidate+0x23/0x112
 [<c018f951>] pid_revalidate+0x3c/0x112
 [<c01777f7>] dput+0x24/0x2ca
 [<c016e3c0>] link_path_walk+0x713/0xb1c
 [<c012d4df>] in_group_p+0x43/0x76
 [<c014042b>] buffered_rmqueue+0xf5/0x239
 [<c018ec72>] proc_info_read+0x53/0x10e
 [<c015d7de>] filp_open+0x5d/0x5f
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 00000001  6440  6838   3115  6902               (NOTLB)
f4b8df34 00000082 c1a11ce0 00000001 0000000f 43c9a065 f4e5c34c f7c3406c 
       00000001 00000001 f7c34038 f7c34058 f220d89c c1a11ce0 00000000 2eb0dc40 
       000f92ad c62aa1b0 c62aa368 00000001 f4b8c000 f4b8df3c fffffe00 c62aa1b0 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

strace        S 396B7140  5848  6902   6838  6903               (NOTLB)
d6ceff4c 00000086 f7292750 396b7140 000f92ad 00017d0e 00000000 c172f1c0 
       ff8085b0 d6ceff30 396b7140 000f92ad f7292750 c1a21ce0 00000000 396b7140 
       000f92ad f66fc230 f66fc3e8 00000282 f7292750 396b7140 fffffe00 f66fc230 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0105e25>] sysenter_past_esp+0x52/0x71

ps            D 00000001  6656  6903   6902                     (NOTLB)
f4b57db8 00000086 c1a21ce0 00000001 0000000f 00000001 00000000 00000002 
       c01162b0 f4b57dd8 c0106979 f4b57d88 00000002 c1a21ce0 00000000 396b7140 
       000f92ad f7292750 f7292908 dbe13132 00000000 f4b57de8 f6fa0b98 f7292750 
Call Trace:
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c0106979>] error_code+0x2d/0x38
 [<c01c71e9>] rwsem_down_failed_common+0xa4/0x17e
 [<c014042b>] buffered_rmqueue+0xf5/0x239
 [<c02cc3f8>] rwsem_down_read_failed+0x29/0x32
 [<c0192bde>] .text.lock.array+0x5b/0xcd
 [<c011944f>] __wake_up_common+0x38/0x57
 [<c0128889>] __group_send_sig_info+0x59/0xd2
 [<c01293e6>] __wake_up_parent+0x37/0x6f
 [<c014042b>] buffered_rmqueue+0xf5/0x239
 [<c018ec72>] proc_info_read+0x53/0x10e
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e77>] syscall_call+0x7/0xb

bash          S 2BE3BBC0  5480  6904   3116  6972               (NOTLB)
d6a19f34 00000086 eae99190 2be3bbc0 000f941c 4f22f065 cba29398 c7c2abac 
       00000001 00000001 2be3bbc0 000f941c eae99190 c1a11ce0 00000000 2be3bbc0 
       000f941c f76b0650 f76b0808 00000001 d6a18000 d6a19f3c fffffe00 f76b0650 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

emacs         T 7E70A840  5340  6972   6904  7026    7092       (NOTLB)
f4b8fe98 00000082 f76b0650 7e70a840 000f941b 00000000 00000086 00000800 
       f71a3084 f7ff5080 7e70a840 000f941b f76b0650 c1a21ce0 00000000 7e70a840 
       000f941b f7d41310 f7d414c8 00000082 00000014 f71a3f24 f6f9356c f7c83a58 
Call Trace:
 [<c012983e>] finish_stop+0x5e/0xce
 [<c012991e>] do_signal_stop+0x70/0x2fa
 [<c011f446>] will_become_orphaned_pgrp+0x20/0x98
 [<c0129e56>] get_signal_to_deliver+0x2ae/0x4f6
 [<c0128ac7>] __kill_pg_info+0x53/0x84
 [<c0105c12>] do_signal+0x6c/0xf1
 [<c01f9d02>] tty_ioctl+0x3b6/0x4c6
 [<c01f9d02>] tty_ioctl+0x3b6/0x4c6
 [<c01729f6>] sys_ioctl+0x104/0x2cd
 [<c012f1ec>] find_task_by_pid+0xd/0x20
 [<c012cc1b>] sys_setpgid+0x62/0x1cc
 [<c0105cd2>] do_notify_resume+0x3b/0x3d
 [<c0105ec2>] work_notifysig+0x13/0x15

bash          S F10C4200  6440  7026   6972  7056               (NOTLB)
d6b03f34 00000086 eae980b0 f10c4200 000f937c 00000000 f5c7a3e4 f5c7a3dc 
       00000001 00000001 f10c4200 000f937c eae980b0 c1a19ce0 00000000 f10c4200 
       000f937c f6f085d0 f6f08788 00010800 d6b02000 d6b03f3c fffffe00 f6f085d0 
Call Trace:
 [<c01215ed>] sys_wait4+0x19b/0x25c
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01c8e86>] copy_to_user+0x50/0x5e
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01216d5>] sys_waitpid+0x27/0x2b
 [<c0105e25>] sysenter_past_esp+0x52/0x71

gdb           S 00000001  5416  7056   7026                     (NOTLB)
d6f2bef8 00000086 c1a19ce0 00000001 0000000f 00000010 00000000 000000d0 
       00000006 00000246 d4819d98 d6f2bfa0 00000246 c1a19ce0 00000000 62045d80 
       000f93a5 eae980b0 eae98268 d6f2beec c01ff066 e5ba0000 00000000 7fffffff 
Call Trace:
 [<c01ff066>] pty_write_room+0x2c/0x2e
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c01f8f56>] tty_poll+0x69/0x77
 [<c0173ad7>] do_pollfd+0x57/0x98
 [<c0173bb2>] do_poll+0x9a/0xb9
 [<c0173d12>] sys_poll+0x141/0x1fc
 [<c0173127>] __pollwait+0x0/0xac
 [<c0105e25>] sysenter_past_esp+0x52/0x71

ps            D 00000001  6440  7092   6904                6972 (NOTLB)
f6b1fdb8 00000086 c1a11ce0 00000001 0000000f f6b1fd88 00000000 00000002 
       c01162b0 f6b1fdd8 c0106979 f6b1fd88 00000002 c1a11ce0 004c4b40 2d613400 
       000f941c eae99190 eae99348 c0324e80 f6b1fdd0 00000096 f6fa0b98 eae99190 
Call Trace:
 [<c01162b0>] do_page_fault+0x0/0x542
 [<c0106979>] error_code+0x2d/0x38
 [<c01c71e9>] rwsem_down_failed_common+0xa4/0x17e
 [<c014042b>] buffered_rmqueue+0xf5/0x239
 [<c02cc3f8>] rwsem_down_read_failed+0x29/0x32
 [<c0192bde>] .text.lock.array+0x5b/0xcd
 [<c018f938>] pid_revalidate+0x23/0x112
 [<c018f951>] pid_revalidate+0x3c/0x112
 [<c01777f7>] dput+0x24/0x2ca
 [<c016e3c0>] link_path_walk+0x713/0xb1c
 [<c012d4df>] in_group_p+0x43/0x76
 [<c014042b>] buffered_rmqueue+0xf5/0x239
 [<c018ec72>] proc_info_read+0x53/0x10e
 [<c015d7de>] filp_open+0x5d/0x5f
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

bash          S 00000001  6440  7093   3117  7182               (NOTLB)
cc165e30 00000086 c1a21ce0 00000001 0000000f f7561000 cc165e58 cc165dfc 
       c0205b06 00000282 00000001 f7561000 cc165e0c c1a21ce0 00000000 3052cb80 
       000f9465 c62aacf0 c62aaea8 00000000 cc165e58 00000010 f7561000 7fffffff 
Call Trace:
 [<c0205b06>] set_cursor+0x75/0x8c
 [<c02cc340>] schedule_timeout+0xbd/0xbf
 [<c0261846>] vgacon_cursor+0x10a/0x26e
 [<c01fdc30>] read_chan+0x861/0xa1f
 [<c01fdf41>] write_chan+0x153/0x219
 [<c0119405>] default_wake_function+0x0/0x12
 [<c0119405>] default_wake_function+0x0/0x12
 [<c01f76a1>] tty_read+0x16a/0x1ab
 [<c015e77e>] vfs_read+0xa1/0x10c
 [<c012a29b>] sys_rt_sigprocmask+0x97/0x140
 [<c015e9e8>] sys_read+0x3f/0x5d
 [<c0105e25>] sysenter_past_esp+0x52/0x71

emacs         T 00000001  6032  7182   7093                     (NOTLB)
ce3a3e98 00000086 c1a11ce0 00000001 0000000f 00000000 00000086 00000800 
       f71a3084 f7ff5080 ce3a3e90 00000082 f7ff5080 c1a11ce0 000f4240 6235e700 
       000f9464 f508f390 f508f548 00000082 00000014 f71a3744 f687c1d0 f7d76540 
Call Trace:
 [<c012983e>] finish_stop+0x5e/0xce
 [<c012991e>] do_signal_stop+0x70/0x2fa
 [<c011f446>] will_become_orphaned_pgrp+0x20/0x98
 [<c0129e56>] get_signal_to_deliver+0x2ae/0x4f6
 [<c0128ac7>] __kill_pg_info+0x53/0x84
 [<c0105c12>] do_signal+0x6c/0xf1
 [<c01f9d02>] tty_ioctl+0x3b6/0x4c6
 [<c01f9d02>] tty_ioctl+0x3b6/0x4c6
 [<c01729f6>] sys_ioctl+0x104/0x2cd
 [<c01078ae>] do_nmi+0x56/0x58
 [<c0105cd2>] do_notify_resume+0x3b/0x3d
 [<c0105ec2>] work_notifysig+0x13/0x15

Attachment: .config
Description: .config