Re: [PATCH] writeback: fix writeback cache thrashing

From: Jan Kara
Date: Wed Jan 02 2013 - 08:43:27 EST

Next message: Mark Brown: "Re: [PATCH] vfs: Add missing errno.h inclusion to namei.h"
Previous message: Jeff Layton: "Re: [PATCH] vfs: Add missing errno.h inclusion to namei.h"
Next in thread: Namjae Jeon: "Re: [PATCH] writeback: fix writeback cache thrashing"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Tue 01-01-13 08:51:04, Wanpeng Li wrote:
> On Mon, Dec 31, 2012 at 12:30:54PM +0100, Jan Kara wrote:
> >On Sun 30-12-12 14:59:50, Namjae Jeon wrote:
> >> From: Namjae Jeon <namjae.jeon@xxxxxxxxxxx>
> >>
> >> Consider Process A: huge I/O on sda
> >> doing heavy write operation - dirty memory becomes more
> >> than dirty_background_ratio
> >> on HDD - flusher thread flush-8:0
> >>
> >> Consider Process B: small I/O on sdb
> >> doing while [1]; read 1024K + rewrite 1024K + sleep 2sec
> >> on Flash device - flusher thread flush-8:16
> >>
> >> As Process A is a heavy dirtier, dirty memory becomes more
> >> than dirty_background_thresh. Due to this, below check becomes
> >> true(checking global_page_state in over_bground_thresh)
> >> for all bdi devices(even for very small dirtied bdi - sdb):
> >>
> >> In this case, even small cached data on 'sdb' is forced to flush
> >> and writeback cache thrashing happens.
> >>
> >> When we added debug prints inside above 'if' condition and ran
> >> above Process A(heavy dirtier on bdi with flush-8:0) and
> >> Process B(1024K frequent read/rewrite on bdi with flush-8:16)
> >> we got below prints:
> >>
> >> [Test setup: ARM dual core CPU, 512 MB RAM]
> >>
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56064 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56704 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84720 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94720 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 384 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 960 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92160 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 256 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 768 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 256 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 320 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92032 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91968 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 1024 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 576 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84352 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 512 KB
> >> [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92608 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92544 KB
> >>
> >> As mentioned in above log, when global dirty memory > global background_thresh
> >> small cached data is also forced to flush by flush-8:16.
> >>
> >> If removing global background_thresh checking code, we can reduce cache
> >> thrashing of frequently used small data.
> > It's not completely clear to me:
> > Why is this a problem? Wearing of the flash? Power consumption? I'd like
> >to understand this before changing the code...
> >
> >> And It will be great if we can reserve a portion of writeback cache using
> >> min_ratio.
> >>
> >> After applying patch:
> >> $ echo 5 > /sys/block/sdb/bdi/min_ratio
> >> $ cat /sys/block/sdb/bdi/min_ratio
> >> 5
> >>
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56064 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56704 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84160 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 96960 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94080 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93120 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93120 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91520 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 89600 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93696 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93696 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 72960 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90624 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90624 KB
> >> [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90688 KB
> >>
> >> As mentioned in the above logs, once cache is reserved for Process B,
> >> and patch is applied there is less writeback cache thrashing on sdb
> >> by frequent forced writeback by flush-8:16 in over_bground_thresh.
> >>
> >> After all, small cached data will be flushed by periodic writeback
> >> once every dirty_writeback_interval.
> > OK, in principle something like this makes sence to me. But if there are
> >more BDIs which are roughly equally used, it could happen none of them are
> >over threshold due to percpu counter & rounding errors. So I'd rather
> >change the conditions to something like:
> > reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
> > bdi_bground_thresh = bdi_dirty_limit(bdi, background_thresh);
> >
> > if (reclaimable > bdi_bground_thresh)
> > return true;
> > /*
> > * If global background limit is exceeded, kick the writeback on
> > * BDI if there's a reasonable amount of data to write (at least
> > * 1/2 of BDI's background dirty limit).
> > */
> > if (global_page_state(NR_FILE_DIRTY) +
> > global_page_state(NR_UNSTABLE_NFS) > background_thresh &&
> > reclaimable * 2 > bdi_bground_thresh)
> > return true;
> >
>
> Hi Jan,
>
> If there are enough BDIs and percpu counter of each bdi roughly equally
> used less than 1/2 of BDI's background dirty limit, still nothing will
> be flushed even if over global background_thresh.
Yes, although then the percpu counter error would have to be quite big.
Anyway, we can change the last condition to:
if (global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS) > background_thresh &&
reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh)

That should be safe and for machines with resonable number of CPUs it
should save the wakeup as well.

Honza

> >> Suggested-by: Wanpeng Li <liwanp@xxxxxxxxxxxxxxxxxx>
> >> Signed-off-by: Namjae Jeon <namjae.jeon@xxxxxxxxxxx>
> >> Signed-off-by: Vivek Trivedi <t.vivek@xxxxxxxxxxx>
> >> Cc: Fengguang Wu <fengguang.wu@xxxxxxxxx>
> >> Cc: Jan Kara <jack@xxxxxxx>
> >> Cc: Dave Chinner <dchinner@xxxxxxxxxx>
> >> ---
> >> fs/fs-writeback.c | 4 ----
> >> 1 file changed, 4 deletions(-)
> >>
> >> diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> >> index 310972b..070b773 100644
> >> --- a/fs/fs-writeback.c
> >> +++ b/fs/fs-writeback.c
> >> @@ -756,10 +756,6 @@ static bool over_bground_thresh(struct backing_dev_info *bdi)
> >>
> >> global_dirty_limits(&background_thresh, &dirty_thresh);
> >>
> >> - if (global_page_state(NR_FILE_DIRTY) +
> >> - global_page_state(NR_UNSTABLE_NFS) > background_thresh)
> >> - return true;
> >> -
> >> if (bdi_stat(bdi, BDI_RECLAIMABLE) >
> >> bdi_dirty_limit(bdi, background_thresh))
> >> return true;
> >> --
> >> 1.7.9.5
> >>
> >--
> >Jan Kara <jack@xxxxxxx>
> >SUSE Labs, CR
> >
> >--
> >To unsubscribe, send a message with 'unsubscribe linux-mm' in
> >the body to majordomo@xxxxxxxxxx For more info on Linux MM,
> >see: http://www.linux-mm.org/ .
> >Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
>
--
Jan Kara <jack@xxxxxxx>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Mark Brown: "Re: [PATCH] vfs: Add missing errno.h inclusion to namei.h"
Previous message: Jeff Layton: "Re: [PATCH] vfs: Add missing errno.h inclusion to namei.h"
Next in thread: Namjae Jeon: "Re: [PATCH] writeback: fix writeback cache thrashing"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]