Re: [RFC][PATCH 00/13] percpu rwsem -v2

[Daniel Wagner]

Hi,

I did a sweep over the parameters for posix01. The parameters are number
of processes and number of locks taken per process. In contrast to the
other test, it looks like there is no set which ends a nice stable
result (read low variance). I have tried several things including
pinning down all processes to CPUs to avoid migration. The results
improved slightly but there was still a high variance.

Anyway I have collected some data and I like to share it. Maybe it is
still useful. All numbers here are without the above mentioned pinning.
There are some runs missing (don't know the reason yet) and I didn't let
it run till the end. So add some salt to these numbers.

The test script and raw data can be found here:

http://monom.org/posix01/

The tables reads:
  nproc: number of process started
  columns: number of locks taken per process

Hardware
  4x E5-4610, for this test all process are scheduled on one socket

First the numbers for tip 4.1.0-02756-ge3d06bd.

nproc 8
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      0.075449     0.210547     0.340658     0.464083     0.590400
std       0.015550     0.024989     0.032080     0.043803     0.055003
min       0.021643     0.067456     0.211779     0.279643     0.327628
25%       0.065337     0.195664     0.318114     0.430040     0.546488
50%       0.075345     0.209411     0.338512     0.461397     0.591433
75%       0.084725     0.226517     0.364190     0.494638     0.626532
max       0.127050     0.281836     0.454558     0.607559     0.762149


nproc 16
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      1.023660     2.463384     3.891954     5.312716     6.752857
std       0.105065     0.124916     0.136476     0.172906     0.207449
min       0.351199     1.527379     3.106403     4.157478     5.519601
25%       0.961098     2.397597     3.807098     5.201875     6.633034
50%       1.031460     2.467317     3.895824     5.321227     6.757502
75%       1.093412     2.539284     3.985122     5.432336     6.889859
max       1.278603     2.785901     4.369434     5.798982     7.324263


nproc 24
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      3.460166     7.942193    11.898540    11.150066    11.060036
std       0.191564     0.232989     0.612868     0.680323     0.465967
min       2.748545     6.575510     9.977165     9.209685     8.937682
25%       3.325521     7.806847    11.440580    10.774070    10.912302
50%       3.493138     7.951859    11.852556    11.163595    11.074910
75%       3.596927     8.088036    12.443429    11.365197    11.243125
max       3.974884     8.589840    13.079780    16.341043    14.244954


nproc 32
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      6.797286    13.943421    14.373278    15.857103    20.047039
std       0.366013     0.417859     0.625967     0.377463     0.302939
min       3.323312    12.266006    12.492706    14.451931    17.496059
25%       6.649401    13.719397    14.186790    15.738348    19.958001
50%       6.868362    13.862458    14.312992    15.870438    20.083564
75%       6.995801    14.027167    14.429383    15.984881    20.215722
max       7.369007    15.631300    21.587450    19.364991    20.755793


nproc 40
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     11.156514    16.936808    18.930412    25.605206    32.334239
std       0.613158     0.614545     0.485336     0.344226     0.398747
min       5.609261    13.147398    16.930261    23.448985    28.992899
25%      10.999876    16.740775    18.788180    25.481274    32.188020
50%      11.251502    16.883100    18.946506    25.648879    32.369347
75%      11.439205    17.032133    19.105678    25.806715    32.565019
max      12.155905    24.116348    26.152117    26.502637    33.263763


nproc 48
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     16.523705    18.214558    27.877811    37.703763    47.655792
std       0.974732     1.118383     0.357481     0.435081     0.472945
min       7.909358    16.279568    25.989797    35.308061    45.279940
25%      16.385582    17.960832    27.729399    37.555420    47.458123
50%      16.692900    18.137635    27.920459    37.767064    47.679325
75%      16.927355    18.311502    28.092018    37.950782    47.926311
max      17.720374    35.810409    28.721941    38.746273    49.333097


nproc 56
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     11.567668    25.100333    38.603884    52.135564    65.716669
std       0.320771     0.369833     0.554834     0.534120     0.612844
min      10.123811    22.598875    35.668780    49.182148    62.504962
25%      11.394438    24.925338    38.389200    51.885988    65.441492
50%      11.593920    25.135043    38.641839    52.206010    65.771692
75%      11.789101    25.328558    38.895343    52.451819    66.068270
max      12.319346    25.948404    46.458428    53.605888    67.270679


nproc 64
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     15.421295    33.254418    51.073912    68.936111    86.919074
std       0.398493     0.411222     0.551629     0.690891     0.694183
min      13.269859    30.900978    48.174802    65.549282    83.099271
25%      15.203732    33.037478    50.821702    68.619365    86.579749
50%      15.467885    33.279869    51.130972    69.001664    86.953804
75%      15.694466    33.514712    51.380860    69.361632    87.341084
max      16.347321    34.475095    52.507292    70.884752    88.807083


nproc 72
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     19.762286    42.488827    65.167763    87.903430   110.666679
std       0.483660     0.480269     0.689872     0.828354     0.892759
min      15.506067    39.937453    61.196633    84.227403   107.014850
25%      19.519194    42.261548    64.834133    87.515837   110.225142
50%      19.809986    42.541263    65.265768    87.974049   110.747980
75%      20.083315    42.792858    65.603762    88.392599   111.223192
max      20.913434    43.830009    66.791452    90.184550   113.062344


nproc 80
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     24.782285    52.853068    80.902314   109.112294   137.441640
std       0.523731     0.639160     0.799033     0.952619     1.091478
min      20.126615    47.813274    77.357915   104.033857   131.978443
25%      24.498501    52.547855    80.509926   108.606293   136.877050
50%      24.835766    52.918841    80.950773   109.197236   137.498470
75%      25.137887    53.244013    81.376380   109.723791   138.101133
max      26.161997    54.372957    83.266046   111.709888   140.419400


nproc 88
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     30.196867    64.467080    98.710365   133.024282   167.330900
std       0.749476     0.691460     0.863908     1.033780     1.240237
min      16.647491    60.034797    94.053510   128.281171   161.778166
25%      29.896764    64.121607    98.290368   132.484092   166.711172
50%      30.271808    64.514222    98.742714   133.089852   167.429483
75%      30.627200    64.903154    99.262584   133.706735   168.086624
max      31.806051    66.343856   101.077264   136.143873   170.449596


nproc 96
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     36.304100    77.194851   117.958001   158.820159   199.868940
std       0.712442     0.718565     1.009163     1.220813     1.462219
min      31.128111    73.850226   112.075970   152.910227   192.977453
25%      35.928427    76.811233   117.466922   158.151278   199.058411
50%      36.378220    77.209148   117.998878   158.879704   199.861157
75%      36.761744    77.636286   118.615380   159.583272   200.701769
max      38.069263    79.445286   120.878239   162.826438   206.826424


nproc 104
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     42.731401    90.887253   138.815476   186.824953   235.055458
std       1.045572     0.742232     0.999065     1.298818     1.554890
min      23.734733    87.384048   133.462821   180.971966   227.475939
25%      42.353032    90.441055   138.213962   186.109237   234.169575
50%      42.861112    90.900274   138.836083   186.835884   235.084204
75%      43.236527    91.382487   139.460129   187.694247   236.011148
max      44.600281    93.394394   141.959512   190.171221   239.491909


nproc 112
              100         200         300         400
count  460.000000  460.000000  460.000000  460.000000
mean    49.782729  105.468739  161.416099  217.385757
std      0.904312    1.011980    1.222772    1.475225
min     45.334285  100.711113  156.087707  210.639527
25%     49.394518  104.971028  160.743875  216.590612
50%     49.906665  105.604756  161.528712  217.437408
75%     50.363428  106.088852  162.187166  218.286111
max     51.800116  108.372299  164.614385  221.788613


And now the same tests for tip+percpu_rwsem:

nproc 8
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      0.285784     0.639623     0.935062     1.165287     1.457565
std       0.040458     0.089317     0.112704     0.094596     0.110337
min       0.118961     0.253775     0.351943     0.869095     1.026194
25%       0.263250     0.600806     0.858630     1.100281     1.376566
50%       0.287019     0.649395     0.930437     1.167166     1.461235
75%       0.312601     0.692013     1.013786     1.228887     1.533511
max       0.407264     0.860837     1.298671     1.460842     1.927867


nproc 16
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      2.338683     5.219408     8.117279    11.050641    14.035433
std       0.146102     0.270400     0.392875     0.510692     0.576044
min       1.836110     4.179970     6.491748     8.998336    11.442838
25%       2.239374     5.042915     7.860587    10.728740    13.667630
50%       2.335801     5.217732     8.125243    11.052183    14.010561
75%       2.443152     5.404223     8.396037    11.404375    14.417740
max       2.798029     5.927344     9.172875    12.203548    15.444552


nproc 24
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean      6.399927    13.673487    20.729554    27.316864    34.125202
std       0.558388     1.157996     1.647191     2.066864     2.487975
min       4.961608    10.767524    17.145018    22.441426    28.566438
25%       5.987118    12.849801    19.555979    25.943463    32.399122
50%       6.388215    13.583983    20.533054    27.122120    33.959403
75%       6.915310    14.786835    22.252796    29.187176    36.308254
max       7.405319    15.823960    23.858206    31.754922    38.997955


nproc 32
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     11.973832    24.885823    36.705614    48.036525    57.418669
std       1.270516     2.604583     3.963139     5.283237     6.441122
min       9.395066    19.958662    27.768684    38.247046    46.265231
25%      10.955417    22.708953    33.510437    43.613011    51.901209
50%      11.801515    24.556642    35.805816    47.315635    55.933447
75%      13.294692    27.520679    40.689642    53.139912    63.860584
max      14.217272    29.968337    44.409489    58.246754    71.045867


nproc 40
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     19.307414    39.204462    55.768040    70.808627    83.830246
std       2.189803     3.982241     5.467692     6.737372     8.124025
min      14.450258    30.606836    44.342114    55.520218    64.704178
25%      17.418113    35.968251    51.341042    65.352697    77.744806
50%      19.067713    39.023460    55.548934    70.282785    83.374667
75%      21.479466    42.666118    60.379906    76.604241    91.158904
max      23.687483    47.019928    67.143361    85.084045   100.957011


nproc 48
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     28.386773    55.462523    77.886706    92.579064   104.319703
std       3.231688     6.142373     8.633285    10.950222    12.510504
min      21.703659    42.486864    56.904221    66.605689    76.529646
25%      25.635256    50.575642    71.306694    82.931995    94.222776
50%      28.136694    55.235674    77.298409    91.993559   104.909015
75%      31.484979    60.645302    85.693462   102.195018   114.141212
max      35.713537    68.342796    96.065304   115.926497   130.916876


nproc 56
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     39.037206    74.470404    97.900979   111.320283   135.943281
std       4.594741     8.940246    11.715321    13.823450    16.032080
min      29.532559    55.193557    65.590273    79.580482    98.565733
25%      35.212004    66.990273    88.066459   100.643871   122.864654
50%      38.796902    73.928176    96.771490   110.669216   136.199617
75%      43.154846    82.041731   108.937264   120.727216   147.769269
max      49.215714    92.181542   125.188702   141.113117   170.961264


nproc 64
               100          200          300          400          500
count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000
mean     51.099012    93.028015   114.649700   145.944300   178.043572
std       6.310777    12.719401    14.675830    18.019135    21.084448
min      36.770938    54.620852    80.837116    98.765936   126.207980
25%      45.955694    84.078285   103.452854   132.127548   160.746493
50%      50.275929    93.031565   114.333533   144.951788   177.105994
75%      56.955477   104.656181   128.418118   163.865640   197.275452
max      63.369715   120.360706   146.542148   182.482159   218.814651


nproc 72
              100         200         300         400         500
count  506.000000  506.000000  506.000000  506.000000  506.000000
mean    64.905270  108.760098  138.811285  179.277895  222.584001
std      8.784532   16.293281   18.160401   21.203767   25.904456
min     43.035451   64.762288   96.401934  127.995159  162.341026
25%     58.658290   98.438247  126.035692  162.944645  202.228444
50%     64.756854  109.608197  139.190635  181.413255  223.359111
75%     72.488483  123.608470  152.745541  195.549278  245.454358
max     83.424516  139.214509  172.538610  218.677815  270.799895


nproc 80
             100         200         300         400         500
count  61.000000   61.000000   61.000000   61.000000   61.000000
mean   76.727789  124.438489  174.095378  225.855798  272.416390
std     9.757928   18.034325   20.216132   24.868596   29.384832
min    55.988043   83.842137  130.842940  173.596051  208.508169
25%    69.218268  116.679810  162.149179  207.015727  252.194955
50%    75.392969  125.378519  173.117425  225.071270  276.188038
75%    83.748328  136.689138  192.392097  245.019530  296.407232
max    97.004966  165.172805  206.391629  266.751069  318.089290


nproc 88
              100
count  157.000000
mean    90.337638
std     15.239911
min     53.393662
25%     79.648088
50%     91.075065
75%    103.530939
max    120.680507


And an attempt at visualization:

http://monom.org/posix01/sweep-4.1.0-02756-ge3d06bd.png
http://monom.org/posix01/sweep-4.1.0-02769-g6ce2591.png


Let me know if these numbers help or not. I start to get better in
running those tests tough they take quite some time to finish. So if
they are useless I sleep well without doing this :)

cheers,
daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/