Re: [PATCHSET v5] Make background writeback great again for the first time

From: Jens Axboe
Date: Fri May 13 2016 - 14:29:22 EST

On 05/11/2016 10:36 AM, Jan Kara wrote:
On Tue 03-05-16 14:17:19, Jan Kara wrote:
The question remains how common a pattern where throttling of background
writeback delays also something else is. I'll schedule a couple of
benchmarks to measure impact of your patches for a wider range of workloads
(but sadly pretty limited set of hw). If ext3 is the only one seeing
issues, I would be willing to accept that ext3 takes the hit since it is
doing something rather stupid (but inherent in its journal design) and we
have a way to deal with this either by enabling delayed allocation or by
turning off the writeback throttling...

So I've run some benchmarks on a machine with 6 GB of RAM and SSD with
queue depth 32. The filesystem on the disk was XFS this time. I've found
couple of regressions. A clear one is with dbench (version 4). The average
throughput numbers look like:

Baseline WBT
Hmean mb/sec-1 30.26 ( 0.00%) 18.67 (-38.28%)
Hmean mb/sec-2 40.71 ( 0.00%) 31.25 (-23.23%)
Hmean mb/sec-4 52.67 ( 0.00%) 46.83 (-11.09%)
Hmean mb/sec-8 69.51 ( 0.00%) 64.35 ( -7.42%)
Hmean mb/sec-16 91.07 ( 0.00%) 86.46 ( -5.07%)
Hmean mb/sec-32 115.10 ( 0.00%) 110.29 ( -4.18%)
Hmean mb/sec-64 145.14 ( 0.00%) 134.97 ( -7.00%)
Hmean mb/sec-512 93.99 ( 0.00%) 133.85 ( 42.41%)

There were also some losses in a filebench webproxy workload (I can give
you exact details of the settings if you want to reproduce it).

Also, and this really puzzles me, I've seen higher read latencies in some
cases (I've verified they are not just noise by rerunning the test for
kernel with writeback throttling patches). For example with the following
fio job file:

[global]
direct=0
ioengine=sync
runtime=300
time_based
invalidate=1
blocksize=4096
size=10g # Just random value, we are running time based workload
log_avg_msec=10
group_reporting=1

[writer]
nrfiles=1
filesize=1g
fdatasync=256
readwrite=randwrite
numjobs=4

[reader]
# Simulate random reading from different files, switching to different file
# after 16 ios. This somewhat simulates application startup.
new_group
filesize=100m
nrfiles=20
file_service_type=random:16
readwrite=randread

I get the following results:

Throughput Baseline WBT
Hmean kb/sec-writer-write 591.60 ( 0.00%) 507.00 (-14.30%)
Hmean kb/sec-reader-read 211.81 ( 0.00%) 137.53 (-35.07%)

So both read and write throughput have suffered. And latencies don't offset
for the loss either:

FIO read latency
Min latency-read 1383.00 ( 0.00%) 1519.00 ( -9.83%)
1st-qrtle latency-read 3485.00 ( 0.00%) 5235.00 (-50.22%)
2nd-qrtle latency-read 4708.00 ( 0.00%) 15028.00 (-219.20%)
3rd-qrtle latency-read 10286.00 ( 0.00%) 57622.00 (-460.20%)
Max-90% latency-read 195834.00 ( 0.00%) 167149.00 ( 14.65%)
Max-93% latency-read 273145.00 ( 0.00%) 200319.00 ( 26.66%)
Max-95% latency-read 335434.00 ( 0.00%) 220695.00 ( 34.21%)
Max-99% latency-read 537017.00 ( 0.00%) 347174.00 ( 35.35%)
Max latency-read 991101.00 ( 0.00%) 485835.00 ( 50.98%)
Mean latency-read 51282.79 ( 0.00%) 49953.95 ( 2.59%)

So we have reduced the extra high read latencies which is nice but on
average there is no change.

And another fio jobfile which doesn't look great:

[global]
direct=0
ioengine=sync
runtime=300
blocksize=4096
invalidate=1
time_based
ramp_time=5 # Let the flusher thread start before taking measurements
log_avg_msec=10
group_reporting=1

[writer]
nrfiles=1
filesize=$((MEMTOTAL_BYTES*2))
readwrite=randwrite

[reader]
# Simulate random reading from different files, switching to different file
# after 16 ios. This somewhat simulates application startup.
new_group
filesize=100m
nrfiles=20
file_service_type=random:16
readwrite=randread

The throughput numbers look like:
Hmean kb/sec-writer-write 24707.22 ( 0.00%) 19912.23 (-19.41%)
Hmean kb/sec-reader-read 886.65 ( 0.00%) 905.71 ( 2.15%)

So we've got significant hit in writes not really offset by a big increase
in reads. Read latency numbers look like (I show the WBT numbers for two runs
just so that one can see how variable the latency numbers are because I was
puzzled by very high max latency for WBT kernels - quartiles seem rather
stable higher percentiles and min/max are rather variable):

Baseline WBT WBT
Min latency-read 1230.00 ( 0.00%) 1560.00 (-26.83%) 1100.00 ( 10.57%)
1st-qrtle latency-read 3357.00 ( 0.00%) 3351.00 ( 0.18%) 3351.00 ( 0.18%)
2nd-qrtle latency-read 4074.00 ( 0.00%) 4056.00 ( 0.44%) 4022.00 ( 1.28%)
3rd-qrtle latency-read 5198.00 ( 0.00%) 5145.00 ( 1.02%) 5095.00 ( 1.98%)
Max-90% latency-read 6594.00 ( 0.00%) 6370.00 ( 3.40%) 6130.00 ( 7.04%)
Max-93% latency-read 11251.00 ( 0.00%) 9410.00 ( 16.36%) 6654.00 ( 40.86%)
Max-95% latency-read 14769.00 ( 0.00%) 13231.00 ( 10.41%) 10306.00 ( 30.22%)
Max-99% latency-read 27826.00 ( 0.00%) 28728.00 ( -3.24%) 25077.00 ( 9.88%)
Max latency-read 80202.00 ( 0.00%) 186491.00 (-132.53%) 141346.00 (-76.24%)
Mean latency-read 5356.12 ( 0.00%) 5229.00 ( 2.37%) 4927.23 ( 8.01%)

I have run also other tests but they have mostly shown no significant
difference.

Thanks Jan, this is great and super useful! I'm revamping certain parts of it to deal with write back caching better, and I'll take a look at the regressions that you reported.

What kind of SSD is this? I'm assuming it's SATA (QD=32), and then it would probably be a safe assumption that it's flagging itself as having a volatile write back cache, would that be a correct assumption?

Are you using scsi-mq, or do you have an IO scheduler attached to it?

--
Jens Axboe