NFS v3 client random-writes slowdown since around 2.6.17 ?

[David Le Corfec]

While investigating a strange MySQL performance regression (amongst an
overall improvement) going from an older hardware running SLES10 SP1
(2.6.16.54) to a newer one running RHEL5 (2.6.18-274), I noticed that
random NFS v3 writes in "big" (2+ GB) files were apparently slower in
"recent" kernels. Stracing MySQL showed me the time taken by the
pwrite() calls was much different between the two OSes.

Stracing a simple pwrite() loop using random offsets, write to the
same NFS volume (on a NetApp server), I had that kind of value for a
single call to pwrite on physical x86_64 servers:
* CentOS 4.9 (2.6.9-100): < 0.02 ms
* SLES10 SP1 (2.6.16-54): < 0.02 ms
* RHEL 5.7 (2.6.18-274): > 0.2 ms
* CentOS 6.0 (2.6.32.?): > 0.2 ms
* openSUSE 11.4 (2.6.37.1): > 0.2 ms

The time taken by the calls in the 2.6.18+ kernels can sometimes even
be greater than 1 ms.
You can find my test program here: http://pastebin.com/VY4WhQPj

I didn't see any problem with sequential writes, random reads, or a
local ext3/4 volume. I tried to vary hardware and nfs server but it
didn't have an noticeable effect.
Mount options: rw,noatime,nolock,noacl,tcp,nfsvers=3,bg,hard,intr,rsize=32768,wsize=32768

Has someone any information/pointer about what happened ? Is it
related to the various NFS and SUNRPC counters added in 2.6.17 ?

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/