Re: [PATCH 00/45] some writeback experiments
From: Wu Fengguang
Date: Thu Oct 08 2009 - 01:35:21 EST
On Wed, Oct 07, 2009 at 11:18:22PM +0800, Wu Fengguang wrote:
> On Wed, Oct 07, 2009 at 09:47:14PM +0800, Peter Staubach wrote:
> >
> > > # vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples)
> > > nr_writeback nr_dirty nr_unstable
> > > 11227 41463 38044
> > > 11227 41463 38044
> > > 11227 41463 38044
> > > 11227 41463 38044
>
> I guess in the above 4 seconds, either client or (more likely) server
> is blocked. A blocked server cannot send ACKs to knock down both
Yeah the server side is blocked. The nfsd are mostly blocked in
generic_file_aio_write(), in particular, the i_mutex lock! I'm copying
one or two big files over NFS, so the i_mutex lock is heavily contented.
I'm using the default wsize=4096 for NFS-root..
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
4693 4693 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
4696 4696 TS - -5 24 1 0.0 D< log_wait_commit nfsd
4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
4693 4693 TS - -5 24 0 0.0 D< sync_buffer nfsd
4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
4696 4696 TS - -5 24 1 0.0 D< generic_file_aio_write nfsd
4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4691 4691 TS - -5 24 0 0.1 D< get_request_wait nfsd
4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd
4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd
4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
4690 4690 TS - -5 24 1 0.1 D< get_write_access nfsd
4691 4691 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4693 4693 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
4694 4694 TS - -5 24 1 0.1 D< get_write_access nfsd
4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
4697 4697 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
Thanks,
Fengguang
> nr_writeback/nr_unstable. And the stuck nr_writeback will freeze
> nr_dirty as well, because the dirtying process is throttled until
> it receives enough "PG_writeback cleared" event, however the bdi-flush
> thread is also blocked when trying to clear more PG_writeback, because
> the client side nr_writeback limit has been reached. In summary,
>
> server blocked => nr_writeback stuck => nr_writeback limit reached
> => bdi-flush blocked => no end_page_writeback() => dirtier blocked
> => nr_dirty stuck
>
> Thanks,
> Fengguang
>
> > > 11045 53987 6490
> > > 11033 53120 8145
> > > 11195 52143 10886
> > > 11211 52144 10913
> > > 11211 52144 10913
> > > 11211 52144 10913
> > >
> > > btrfs seems to maintain a private pool of writeback pages, which can go out of
> > > control:
> > >
> > > nr_writeback nr_dirty
> > > 261075 132
> > > 252891 195
> > > 244795 187
> > > 236851 187
> > > 228830 187
> > > 221040 218
> > > 212674 237
> > > 204981 237
> > >
> > > XFS has very interesting "bumpy writeback" behavior: it tends to wait
> > > collect enough pages and then write the whole world.
> > >
> > > nr_writeback nr_dirty
> > > 80781 0
> > > 37117 37703
> > > 37117 43933
> > > 81044 6
> > > 81050 0
> > > 43943 10199
> > > 43930 36355
> > > 43930 36355
> > > 80293 0
> > > 80285 0
> > > 80285 0
> > >
> > > Thanks,
> > > Fengguang
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/