Re: 2.6.18 mmap hangs unrelated apps

From: Trond Myklebust
Date: Fri Dec 15 2006 - 14:45:20 EST

On Fri, 2006-12-15 at 11:50 -0600, Michal Sabala wrote:
> On 2006/12/15 at 10:24:15 Trond Myklebust <trond.myklebust@xxxxxxxxxx> wrote
> > On Thu, 2006-12-14 at 20:30 -0600, Michal Sabala wrote:
> > >
> > > `cat /proc/*PID*/wchan` for all hanging processes contains page_sync.
> >
> > Have you tried an 'echo t >/proc/sysrq-trigger' on a client with one of
> > these hanging processes? If so, what does the output look like?
>
> Hello Trond,
>
> Below is the sysrq trace output for XFree86 which entered the
> uninterruptible sleep state on the P4 machine with nfs /home. Please
> note that XFree86 does not have any files open in /home - as reported by
> `lsof`. Below, I also listed the output of vmstat.


It is hanging because it is trying to free up memory by reclaiming pages
that are held by your mmapped file on NFS. Do you know why NFS is
hanging?

Cheers
Trond

> XFree86 D 00000003 0 2471 2453 (NOTLB)
> c4871c0c 00003082 c86b72bc 00000003 cb7c94a4 0000001d 3b67f3ff c0146dd2
> c1184180 cb3e7110 00000000 001ec7ff a60f8097 00000089 c02e1e60 cb3e7000
> c1184180 00000000 c1180030 c4871c18 c028c7d8 c4871c5c c01435b6 c01435f3
> Call Trace:
> [<c0146dd2>] free_pages_bulk+0x1d/0x1d4
> [<c028c7d8>] io_schedule+0x26/0x30
> [<c01435b6>] sync_page+0x0/0x40
> [<c01435f3>] sync_page+0x3d/0x40
> [<c028c9ce>] __wait_on_bit_lock+0x2c/0x52
> [<c0143c13>] __lock_page+0x6a/0x72
> [<c012ec77>] wake_bit_function+0x0/0x3c
> [<c012ec77>] wake_bit_function+0x0/0x3c
> [<c0149d2f>] pagevec_lookup+0x17/0x1d
> [<c014a085>] truncate_inode_pages_range+0x20a/0x260
> [<c014a0e4>] truncate_inode_pages+0x9/0xc
> [<c0172c8a>] generic_delete_inode+0xb6/0x10f
> [<c0172e73>] iput+0x5f/0x61
> [<c01706bd>] dentry_iput+0x68/0x83
> [<c01707d8>] dput+0x100/0x118
> [<ccb6c334>] put_nfs_open_context+0x67/0x88 [nfs]
> [<ccb701ed>] nfs_release_request+0x38/0x47 [nfs]
> [<ccb736dd>] nfs_wait_on_requests_locked+0x62/0x98 [nfs]
> [<ccb74c32>] nfs_sync_inode_wait+0x4a/0x130 [nfs]
> [<ccb6b639>] nfs_release_page+0x0/0x30 [nfs]
> [<ccb6b655>] nfs_release_page+0x1c/0x30 [nfs]
> [<c015f37c>] try_to_release_page+0x34/0x46
> [<c014aa8b>] shrink_page_list+0x263/0x350
> [<c0104db8>] do_IRQ+0x48/0x50
> [<c01036c6>] common_interrupt+0x1a/0x20
> [<c014acd7>] shrink_inactive_list+0x9b/0x248
> [<c014b2fd>] shrink_zone+0xb5/0xd0
> [<c014b382>] shrink_zones+0x6a/0x7e
> [<c014b48e>] try_to_free_pages+0xf8/0x1da
> [<c0147a18>] __alloc_pages+0x17c/0x278
> [<c014f555>] do_anonymous_page+0x45/0x150
> [<c014f9f7>] __handle_mm_fault+0xda/0x1bf
> [<c0115849>] do_page_fault+0x1c4/0x4bc
> [<c01021b7>] restore_sigcontext+0x10c/0x15f
> [<c0115685>] do_page_fault+0x0/0x4bc
> [<c0103809>] error_code+0x39/0x40
>
>
>
>
> $> vmstat
> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 2 1 0 82128 11484 36096 0 0 12 21 311 287 8 3 0 89
>
> $> vmstat -m
> Cache Num Total Size Pages
> nfs_direct_cache 0 0 76 50
> nfs_write_data 36 91 512 7
> nfs_read_data 32 35 512 7
> nfs_inode_cache 28 108 648 6
> nfs_page 1 59 64 59
> rpc_buffers 8 8 2048 2
> rpc_tasks 8 15 256 15
> rpc_inode_cache 8 14 512 7
> fib6_nodes 5 113 32 113
> ip6_dst_cache 4 15 256 15
> ndisc_cache 1 15 256 15
> RAWv6 4 6 640 6
> UDPv6 1 6 640 6
> tw_sock_TCPv6 0 0 128 30
> request_sock_TCPv6 0 0 128 30
> TCPv6 2 3 1280 3
> ip_conntrack_expect 0 0 96 40
> ip_conntrack 13 68 224 17
> ip_fib_alias 9 113 32 113
> ip_fib_hash 9 113 32 113
> jbd_4k 0 0 4096 1
> Cache Num Total Size Pages
> ext3_inode_cache 12284 25352 504 8
> ext3_xattr 0 0 48 78
> journal_handle 2 169 20 169
> journal_head 85 504 52 72
> revoke_table 2 254 12 254
> revoke_record 0 0 16 203
> uhci_urb_priv 1 127 28 127
> clip_arp_cache 0 0 256 15
> UNIX 46 133 512 7
> flow_cache 0 0 128 30
> cfq_ioc_pool 29 84 92 42
> cfq_pool 27 80 96 40
> crq_pool 20 84 44 84
> deadline_drq 0 0 44 84
> as_arq 0 0 56 67
> mqueue_inode_cache 1 6 640 6
> dnotify_cache 0 0 20 169
> dquot 0 0 128 30
> eventpoll_pwq 0 0 36 101
> eventpoll_epi 0 0 128 30
> inotify_event_cache 0 0 28 127
> Cache Num Total Size Pages
> inotify_watch_cache 1 92 40 92
> kioctx 0 0 256 15
> kiocb 0 0 128 30
> fasync_cache 2 203 16 203
> shmem_inode_cache 452 459 448 9
> posix_timers_cache 0 0 88 44
> uid_cache 3 59 64 59
> ip_mrt_cache 0 0 128 30
> tcp_bind_bucket 12 203 16 203
> inet_peer_cache 4 59 64 59
> secpath_cache 0 0 32 113
> xfrm_dst_cache 0 0 384 10
> ip_dst_cache 18 45 256 15
> arp_cache 3 15 256 15
> RAW 2 7 512 7
> UDP 9 14 512 7
> tw_sock_TCP 0 0 128 30
> request_sock_TCP 0 0 64 59
> TCP 12 21 1152 7
> blkdev_ioc 29 127 28 127
> blkdev_queue 19 20 956 4
> Cache Num Total Size Pages
> blkdev_requests 17 23 172 23
> biovec-256 7 8 3072 2
> biovec-128 7 10 1536 5
> biovec-64 7 10 768 5
> biovec-16 7 15 256 15
> biovec-4 7 59 64 59
> biovec-1 51 203 16 203
> bio 276 300 128 30
> sock_inode_cache 73 154 512 7
> skbuff_fclone_cache 1 10 384 10
> skbuff_head_cache 459 615 256 15
> file_lock_cache 40 40 96 40
> Acpi-Operand 699 736 40 92
> Acpi-ParseExt 0 0 44 84
> Acpi-Parse 0 0 28 127
> Acpi-State 0 0 44 84
> Acpi-Namespace 319 338 20 169
> proc_inode_cache 1218 1310 368 10
> sigqueue 16 27 144 27
> radix_tree_node 989 2576 276 14
> bdev_cache 4 7 512 7
> Cache Num Total Size Pages
> sysfs_dir_cache 3656 3696 44 84
> mnt_cache 25 30 128 30
> inode_cache 724 869 352 11
> dentry_cache 8164 26100 132 29
> filp 693 1840 192 20
> names_cache 1 1 4096 1
> key_jar 6 30 128 30
> idr_layer_cache 100 116 136 29
> buffer_head 3614 6192 52 72
> mm_struct 54 81 448 9
> vm_area_struct 1272 2576 84 46
> fs_cache 72 118 64 59
> files_cache 102 135 256 15
> signal_cache 55 90 384 10
> sighand_cache 66 66 1344 3
> task_struct 68 87 1360 3
> anon_vma 647 1524 12 254
> pgd 40 40 4096 1
> pid 65 202 36 101
> size-131072(DMA) 0 0 131072 1
> size-131072 0 0 131072 1
> Cache Num Total Size Pages
> size-65536(DMA) 0 0 65536 1
> size-65536 1 1 65536 1
> size-32768(DMA) 0 0 32768 1
> size-32768 0 0 32768 1
> size-16384(DMA) 0 0 16384 1
> size-16384 0 0 16384 1
> size-8192(DMA) 0 0 8192 1
> size-8192 67 69 8192 1
> size-4096(DMA) 0 0 4096 1
> size-4096 33 33 4096 1
> size-2048(DMA) 0 0 2048 2
> size-2048 503 534 2048 2
> size-1024(DMA) 0 0 1024 4
> size-1024 197 208 1024 4
> size-512(DMA) 0 0 512 8
> size-512 241 296 512 8
> size-256(DMA) 0 0 256 15
> size-256 72 90 256 15
> size-192(DMA) 0 0 192 20
> size-192 308 320 192 20
> size-128(DMA) 0 0 128 30
> Cache Num Total Size Pages
> size-128 455 480 128 30
> size-96(DMA) 0 0 128 30
> size-96 672 720 128 30
> size-64(DMA) 0 0 64 59
> size-32(DMA) 0 0 32 113
> size-64 946 1357 64 59
> size-32 2424 2599 32 113
> kmem_cache 133 150 128 30
>
>
> Thanks, Michal
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/