OK, I ran some tests, bypassing NFS, and got the following vmstat display:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 1 696 80692 844 7225424 0 0 92 263924 14380 6176 2 40 46 12
6 0 696 108668 784 7200936 0 0 456 6444 30515 20641 3 57 36 3
5 1 696 100916 768 7211220 0 0 508 97276 31761 19916 3 65 27 4
7 0 700 91188 764 7217340 0 0 428 121576 28704 21957 3 69 24 3
3 3 700 103356 780 7204472 0 0 408 121748 29513 22603 3 66 23 8
7 0 700 92836 780 7216168 0 0 360 43508 28784 21410 3 54 33 11
7 0 700 88364 768 7224272 0 0 296 158236 26530 17570 3 66 21 10
10 0 700 91068 776 7219096 0 0 444 141456 30306 16053 3 74 16 7
5 2 700 102212 752 7206676 0 0 340 170076 29249 14872 2 69 19 10
11 0 700 87884 768 7222096 0 0 392 143312 29743 19808 1 65 23 11
8 0 700 104692 744 7204644 0 0 240 159624 25814 18747 3 58 20 19
9 0 700 107196 736 7205196 0 0 344 148500 28191 18113 3 70 21 6
16 1 700 99300 768 7211364 0 12 464 164348 25671 18326 4 64 18 15
5 4 700 107412 1052 7204824 0 0 936 170904 28994 17062 3 73 11 14
8 1 700 98892 1284 7217512 0 0 596 182708 31520 18424 1 76 13 10
This ia with 6 concurrent data streams. In addition to the /proc/sys/vm/dirty* values I also adjusted
values in mm/page-writeback.c eg. MAX_WRITEBACK_PAGES to 8192 . Our goal is to blast out
as much as possible per pdflush invocation.