Re: [PATCH] writeback: fix writeback cache thrashing

[Jan Kara]

On Wed 09-01-13 17:26:36, Namjae Jeon wrote:
<snip>
> But in one normal scenario, the changes actually results in
> performance degradation.
> 
> Results for âddâ thread on two devices:
> Before applying Patch:
> #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 &
> #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 &
> #>
> #> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.0GB) copied, 77.205276 seconds, 25.9MB/s  -> USB
> HDD WRITE Speed
> 
> [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000
> #>
> #>
> #> 800+0 records in
> 800+0 records out
> 838860800 bytes (800.0MB) copied, 154.528362 seconds, 5.2MB/s -> USB
> Flash WRITE Speed
> 
> After applying patch:
> #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 &
> dd if=/
> #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 &
> #>
> #> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.0GB) copied, 123.844770 seconds, 16.1MB/s ->USB
> HDD WRITE Speed
> 800+0 records in
> 800+0 records out
> 838860800 bytes (800.0MB) copied, 141.352945 seconds, 5.7MB/s -> USB
> Flash WRITE Speed
> 
> [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000
> [1]+ Done dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800
> 
> So, after applying our changes:
> 1) USB HDD Write speed dropped from 25.9 -> 16.1 MB/s
> 2) USB Flash Write speed increased marginally from 5.2 -> 5.7 MB/s
> 
> Normally if we have a USB Flash and HDD plugged in system. And if we
> initiate the âddâ on both the devices. Once dirty memory is more than
> the background threshold, flushing starts for all BDI (The write-back
> for the devices will be kicked by the condition):
> If (global_page_state(NR_FILE_DIRTY) +
> global_page_state(NR_UNSTABLE_NFS) > background_thresh))
> 	return true;
> As the slow device and the fast device always make sure that there is
> enough DIRTY data in memory to kick write-back.
> Since, USB Flash is slow, the DIRTY pages corresponding to this device
> is much higher, resulting in returning âtrueâ everytime from
> over_bground_thresh. So, even though HDD might have only few KB of
> dirty data, it is also flushed immediately.
> This frequent flushing of HDD data results in gradually increasing the
> bdi_dirty_limit() for HDD.
  Interesting. Thanks for testing! So is this just a problem with initial
writeout fraction estimation. I.e. if you first let dd to USB HDD run for a
couple of seconds to ramp up its fraction and only then start writeout to
USB flash, is there still a problem with USB HDD throughput with the
changed over_bground_thresh() function?

> But, when we introduce the change to control per BDI i.e.,
>  if (global_page_state(NR_FILE_DIRTY) +
>          global_page_state(NR_UNSTABLE_NFS) > background_thresh &&
>          reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh)
> 
> Now, in this case, when we consider the same scenario, writeback for
> HDD will only be kicked only if âreclaimable * 2 + bdi_stat_error(bdi)
> * 2 > bdi_bground_threshâ
> But this condition is not true a lot many number of times, so
> resulting in false.
  I'm surprised it's not true so often... dd(1) should easily fill the
caches. But maybe we are oscilating between below-background-threshold
and at-dirty-limit situations rather quickly. Do you have recordings of
BDI_RECLAIMABLE and BDI_DIRTY from the problematic run?

> This continuous failure to start write-back for HDD actually results
> in lowering the bdi_dirty_limit for HDD, in a way PAUSING the writer
> thread for HDD.
> This is actually resulting in less number of WRITE operations per
> second for HDD. As, the âddâ on USB HDD will be put to long sleep(MAX
> PAUSE) in balance_dirty_pages.
> 
> While for USB Flash, its bdi_dirty_limit is kept on increasing as it
> is getting more chance to flush dirty data in over_bground_thresh. As,
> bdi_reclaimable > bdi_dirty_limit is true. So, resulting more number
> of WRITE operation per second for USB Flash.
> From these observations, we feel that these changes might not be
> needed. Please let us know in case we are missing on any point here,
> we can further check more on this.
  Well, at least we know changing the condition has unexpected side
effects. I'd like to understand those before discarding the idea - because
in your setup flusher thread must end up writing rather small amount of
pages in each run when it's running continuously and that's not too good
either...

								Honza
-- 
Jan Kara <jack@xxxxxxx>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/