Re: Bad SSD performance with recent kernels

From: Wu Fengguang
Date: Sat Jan 28 2012 - 07:56:14 EST


Hi Herbert,

On Fri, Jan 27, 2012 at 07:00:34AM +0100, Herbert Poetzl wrote:
>
> Dear Community!
>
> Recently I decided to update the kernel on a Dell Laptop to
> a more recent version than 2.6.38.x, but experienced bad
> I/O performance with the new kernel, so I started to dig
> a little deeper and ended up with the following test:
>
> 1) download and extract kernel (on 2.6.38.8)
> 2) make defconfig
> 3) make localmodconfig
> 4) make
> 5) make modules_install install
>
> Then I booted each kernel in single user and ran the following
> test script:
>
> echo noop >/sys/class/block/sda/queue/scheduler
> for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
> /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
> ionice -c0 nice -20 \
> dd if=/dev/sda of=/dev/null bs=1M count=20480
>
> echo deadline >/sys/class/block/sda/queue/scheduler
> for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
> /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
> ionice -c0 nice -20 \
> dd if=/dev/sda of=/dev/null bs=1M count=20480
>
> echo cfq >/sys/class/block/sda/queue/scheduler
> for n in 1 2 3; do sync; echo $n > /proc/sys/vm/drop_caches; done
> /usr/bin/time -f "real = %e, user = %U, sys = %S, %P cpu" \
> ionice -c0 nice -20 \
> dd if=/dev/sda of=/dev/null bs=1M count=20480
>
>
> note that the Laptop is a relatively modern Latitude E6400
> with a Samsung 830 Series 256GB SSD
>
> here are the surprising results:
>
> @ linux 2.6.38.8
> 248 MB/s real = 86.74, user = 0.01, sys = 21.65, 24% cpu
> 248 MB/s real = 86.81, user = 0.02, sys = 21.75, 25% cpu
> 251 MB/s real = 85.63, user = 0.01, sys = 22.24, 25% cpu
>
> @ linux 2.6.39.4
> 49.0 MB/s real = 438.79, user = 0.01, sys = 19.79, 4% cpu
> 25.7 MB/s real = 836.70, user = 0.02, sys = 18.39, 2% cpu
> 27.7 MB/s real = 776.53, user = 0.01, sys = 16.03, 2% cpu
>
> @ linux 3.0.18
> 48.9 MB/s real = 439.07, user = 0.01, sys = 17.55, 4% cpu
> 25.0 MB/s real = 859.03, user = 0.01, sys = 16.97, 1% cpu
> 49.8 MB/s real = 431.61, user = 0.01, sys = 16.68, 3% cpu
>
> @ linux 3.1.10
> 54.0 MB/s real = 398.23, user = 0.01, sys = 17.36, 4% cpu
> 29.4 MB/s real = 731.47, user = 0.01, sys = 17.14, 2% cpu
> 25.0 MB/s real = 859.35, user = 0.01, sys = 14.51, 1% cpu
>
> @ linux 3.2.2
> 45.8 MB/s real = 468.85, user = 0.01, sys = 17.11, 3% cpu
> 44.8 MB/s real = 478.92, user = 0.01, sys = 17.02, 3% cpu
> 45.0 MB/s real = 476.91, user = 0.01, sys = 16.14, 3% cpu

Thanks for the through tests! However I cannot reproduce the regressions,
but see slightly better performance in 3.2:

Linux lkp-nex04 3.2.0-rc7-shli+ #121 SMP Thu Jan 19 18:10:45 CST 2012 x86_64 x86_64 x86_64 GNU/Linux

21474836480 bytes (21 GB) copied, 90.9307 s, 236 MB/s real = 91.03, user = 0.01, sys = 25.80, 28% cpu
21474836480 bytes (21 GB) copied, 90.8864 s, 236 MB/s real = 90.90, user = 0.01, sys = 25.54, 28% cpu
21474836480 bytes (21 GB) copied, 93.4684 s, 230 MB/s real = 93.47, user = 0.02, sys = 25.12, 26% cpu


Linux lkp-nex04 2.6.38 #334 SMP Sat Jan 28 20:16:25 CST 2012 x86_64 GNU/Linux

21474836480 bytes (21 GB) copied, 96.2382 s, 223 MB/s real = 96.29, user = 0.01, sys = 57.64, 59% cpu
21474836480 bytes (21 GB) copied, 96.5796 s, 222 MB/s real = 96.60, user = 0.02, sys = 58.08, 60% cpu
21474836480 bytes (21 GB) copied, 95.486 s, 225 MB/s real = 95.51, user = 0.00, sys = 58.20, 60% cpu

My test box has 64GB memory and a dozen 80GB INTEL SSDSA2M080G2GN SSD drives attached to

87:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)

> I have no idea why the I/O performance is that bad on any
> kernel newer than 2.6.38.x, but I'm happy to test and/or
> try various configurations as time permits ...

Would you please create a filesystem and large file on sda and run the
tests on the file? There was some performance bug on reading the raw
/dev/sda device file..

> The detailed test results as well as the dmesg and config
> of each kernel can be found here:
> http://vserver.13thfloor.at/Stuff/SSD

It would be better for such files be attached in future.
(For one thing I get timeout when trying to open the link...)

Thanks,
Fengguang

> here some technical information:
>
> http://www.dell.com/us/dfb/p/latitude-e6400/pd
> SATA controller: Intel Corporation ICH9M/M-E SATA AHCI Controller (rev 03)
>
> http://www.samsung.com/uk/consumer/memory-cards-hdd-odd/ssd/ssd/MZ-7PC256N/EU-spec
> Model=SAMSUNG SSD 830 Series, FwRev=CXM03B1Q
> supposed 520MB/s seq. read, 320MB/s seq. write, 75K IOPS
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/