Re: ppoll() stuck on POLLIN while TCP peer is sending
From: Eric Wong
Date: Fri Jan 04 2013 - 20:07:06 EST
Mel Gorman <mgorman@xxxxxxx> wrote:
> On Wed, Jan 02, 2013 at 08:08:48PM +0000, Eric Wong wrote:
> > Instead, I disabled THP+compaction under v3.7.1 and I've been unable to
> > reproduce the issue without THP+compaction.
> Implying that it's stuck in compaction somewhere. It could be the case
> that compaction alters timing enough to trigger another bug. You say it
> tests differently depending on whether TCP or unix sockets are used
> which might indicate multiple problems. However, lets try and see if
> compaction is the primary problem or not.
I've only managed to encounter this issue with TCP sockets.
No luck reproducing the issue with Unix sockets, not even with 90K
buffers as suggested by Eric Dumazet. This seems unique to TCP.
Fwiw, I also tried going back to a 16K MTU on loopback a few days ago,
but was still able to reproduce the issue, so
commit 0cf833aefaa85bbfce3ff70485e5534e09254773 doesn't seem
to be a culprit, either.
> > As I mention in http://mid.gmane.org/20121229113434.GA13336@xxxxxxxxxxxxx
> > I run my below test (`toosleepy') with heavy network and disk activity
> > for a long time before hitting this.
> Using a 3.7.1 or 3.8-rc2 kernel, can you reproduce the problem and then
> answer the following questions please?
OK, I'm on 3.8-rc2.
> 1. What are the contents of /proc/vmstat at the time it is stuck?
> 2. What are the contents of /proc/PID/stack for every toosleepy
> process when they are stuck?
Oops, I needed a rebuild with CONFIG_STACKTRACE=y (it took some effort
to get the right combination of options).
I probably enabled a few more debugging options than I needed and it
seems to have taken longer to reproduce the issue. Unfortunately I was
distracted when toosleepy got stuck and missed the change to inspect
before hitting ETIMEDOUT :x
Attempting to reproduce the issue while I'm looking.
> 3. Can you do a sysrq+m and post the resulting dmesg?
SysRq : Show Memory
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 0: hi: 186, btch: 31 usd: 144
CPU 1: hi: 186, btch: 31 usd: 160
active_anon:3358 inactive_anon:3379 isolated_anon:0
active_file:10615 inactive_file:92319 isolated_file:0
unevictable:0 dirty:3 writeback:0 unstable:0
free:2240 slab_reclaimable:0 slab_unreclaimable:0
mapped:2333 shmem:114 pagetables:697 bounce:0
DMA free:2408kB min:84kB low:104kB high:124kB active_anon:8kB inactive_anon:44kB active_file:824kB inactive_file:11512kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:112kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve: 0 489 489 489
DMA32 free:6552kB min:2784kB low:3480kB high:4176kB active_anon:13424kB inactive_anon:13472kB active_file:41636kB inactive_file:357764kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500952kB managed:491396kB mlocked:0kB dirty:12kB writeback:0kB mapped:9316kB shmem:456kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:1160kB pagetables:2768kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve: 0 0 0 0
DMA: 52*4kB (UMR) 13*8kB (UMR) 4*16kB (R) 2*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 2424kB
DMA32: 1608*4kB (UM) 15*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6552kB
103053 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 392188kB
Total swap = 392188kB
131054 pages RAM
3477 pages reserved
283467 pages shared
111732 pages non-shared
> What I'm looking for is a throttling bug (if pgscan_direct_throttle is
> elevated), an isolated page accounting bug (nr_isolated_* is elevated
> and process is stuck in congestion_wait in a too_many_isolated() loop)
> or a free page accounting bug (big difference between nr_free_pages and
> buddy list figures).
> I'll try reproducing this early next week if none of that shows an
> obvious candidate.
Thanks! I'll try to get you more information as soon as possible.
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/