Re: BUG: Slowdown on 3000 socket-machines tracked down
From: Christian Schmid
Date: Thu Mar 10 2005 - 14:11:43 EST
Andrew Morton wrote:
Christian Schmid <webmaster@xxxxxxxxxxxxxx> wrote:
> So, maybe a VM problem? That would be a good place to focus since
> I think we can be fairly certain it isn't a problem in just the
> networking code. Otherwise, my tests would show lower bandwidth.
Thanks to your tests I am really sure that its no network-code problem anymore. But what I THINK it
is: The network is allocating buffers dynamically and if the vm doesnt provide that buffers fast
enough, it locks as well.
Did anyone have a 100-liner which demonstrates this problem?
The output of `vmstat 1' when the thing starts happening would be interesting.
There you go. As you can see, free is rather high when lower_zone_protection is set to 1024. When I
set it to 0, free goes down and the slow-down starts when the memory is full. The slow-down goes
away after I set lower_zone_protection to 1024, but only AFTER the free-memory doesnt rise up more.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 34 0 588944 9120 6925452 0 0 29 14 10 12 15 25 27 33
3 30 0 589648 9060 6924628 0 0 68176 1488 4770 6545 12 34 0 53
1 32 0 589712 9152 6924944 0 0 66152 2824 5606 6621 10 34 0 55
3 29 0 590352 8992 6924220 0 0 68260 28 4123 7809 17 36 0 47
1 33 0 601744 8684 6899096 0 0 57776 28 4015 6267 16 38 0 45
6 31 0 602960 8960 6911604 0 0 56148 124 4659 6013 17 36 0 48
7 31 0 590736 8776 6903220 0 0 56460 824 4521 5940 17 35 0 48
0 32 0 589264 9064 6923536 0 0 67376 96 5135 6918 15 34 0 51
0 33 0 590928 8912 6923620 0 0 69504 108 4604 6487 13 34 0 53
3 30 0 589008 9080 6924472 0 0 66904 72 4300 7336 14 35 0 51
1 29 0 590544 9156 6924124 0 0 67684 28 4535 7298 16 34 0 50
2 32 0 589968 9052 6923956 0 0 61000 88 4293 6898 14 35 0 51
1 29 0 591120 8876 6923384 0 0 67940 176 4455 7259 12 34 0 54
3 31 0 589520 9024 6925616 0 0 66980 20 4909 7037 12 32 0 56
4 30 0 590096 8972 6924444 0 0 63924 80 4308 6203 14 33 0 53
9 29 0 590096 8876 6915836 0 0 59860 32 4507 6268 18 36 0 47
0 28 0 588688 9120 6923276 0 0 66844 76 4280 6025 14 32 0 54
3 31 0 581072 9336 6930744 0 0 47744 20788 5307 5195 10 31 0 59
5 28 0 500432 10456 6980352 0 0 62032 20 4749 6779 22 36 0 42
8 31 0 432720 11620 7056504 0 0 64420 4 4758 6480 29 40 0 31
10 30 0 383120 12808 7129232 0 0 72844 0 5197 7040 15 30 0 55
3 30 0 313552 13840 7198716 0 0 69516 0 4479 6216 16 32 0 51
3 28 0 245016 14912 7265508 0 0 67028 192 5111 6295 15 30 0 55
5 26 0 158744 15788 7344396 0 0 78576 0 4361 5937 19 33 0 49
1 28 0 67544 16652 7428668 0 0 84284 0 4299 5252 17 32 0 51
0 32 0 16572 17244 7463436 0 0 83744 2632 4900 6126 16 35 0 49
3 22 0 24196 17616 7453476 0 0 68804 5644 5148 5522 17 34 0 49
3 25 0 24480 17960 7453812 0 0 64872 1668 5043 5568 14 33 0 53
3 26 0 20780 18328 7456300 0 0 64692 0 4854 7081 14 33 0 53
3 32 0 20640 18432 7456672 0 0 60496 28 4882 8156 16 32 0 52
SLOWDOWN START:
2 29 0 21056 18660 7454880 0 0 58432 28 4615 8019 16 33 0 51
7 30 0 24088 18904 7451780 0 0 57920 0 5571 9829 13 32 0 55
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
5 30 0 28340 19228 7448736 0 0 61208 216 5602 9912 12 32 0 56
2 29 0 22680 19416 7453852 0 0 62096 8 6029 10302 13 33 0 54
0 31 0 23136 19616 7454128 0 0 55776 0 5986 10845 14 33 0 53
1 32 0 25240 19136 7452364 0 0 47172 48 6609 10681 18 36 0 46
2 30 0 16988 19004 7461064 0 0 62988 2352 8611 12934 10 29 0 61
0 35 0 26432 18884 7449488 0 0 46032 12920 8253 9289 9 27 0 63
0 35 0 23636 19000 7456104 0 0 44456 164 9576 11174 8 23 0 70
3 29 0 16932 18996 7465152 0 0 60108 12 9062 13320 14 28 1 57
3 31 0 24720 18768 7458648 0 0 60744 24 9926 15865 14 27 0 59
1 33 0 24284 18724 7459848 0 0 63152 0 10028 16689 13 27 0 61
1 31 0 24856 18472 7462208 0 0 59384 0 10157 16561 12 26 1 60
0 34 0 24272 18192 7462556 0 0 60276 184 10946 18029 10 25 0 64
0 34 0 25604 18312 7461756 0 0 58244 0 10217 16344 11 27 0 62
1 29 0 20816 18416 7467296 0 0 61928 0 10796 16894 10 26 0 65
2 33 0 23388 18764 7466744 0 0 47620 0 8889 15021 19 30 0 51
0 34 0 16612 18972 7473540 0 0 54648 12 10644 16752 13 26 1 59
0 35 0 22436 19024 7469068 0 0 55192 2864 11080 17519 12 26 1 61
1 33 0 16548 19192 7474204 0 0 52756 796 11412 19072 12 26 0 62
1 35 0 21352 18904 7470140 0 0 50104 4400 11999 16810 9 23 1 68
1 33 0 24412 18824 7468452 0 0 55132 80 11441 17418 9 25 2 64
1 31 0 24384 18812 7468736 0 0 49860 128 11745 19884 10 26 3 62
5 32 0 27660 19232 7465460 0 0 47396 80 10758 16619 14 27 2 57
2 31 0 31832 19596 7461628 0 0 53480 0 11355 18973 12 27 0 61
2 33 0 31264 19792 7463268 0 0 53236 88 11552 18697 10 26 0 64
2 31 0 31596 20008 7461692 0 0 52624 136 11300 19832 15 27 0 58
2 30 0 22596 20260 7473204 0 0 51864 12 11993 21202 12 25 1 62
0 33 0 20824 20460 7475044 0 0 52488 236 12395 19848 8 24 1 67
2 30 0 21204 20508 7474520 0 0 56616 72 12258 22120 10 25 4 60
3 32 0 16720 20772 7480512 0 0 53972 40 11942 20908 13 26 3 58
3 28 0 31884 20732 7464776 0 0 47236 92 11602 19614 13 26 0 61
0 12 0 16700 20776 7479828 0 0 46936 1416 11557 18017 9 24 10 56
1 31 0 25040 20896 7474268 0 0 38788 6752 10727 14073 12 23 4 61
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 31 0 16748 21080 7481972 0 0 49472 8 12436 19864 10 22 11 57
0 35 0 23588 21240 7475828 0 0 58944 84 12935 22227 8 25 0 66
1 30 0 24516 21260 7475128 0 0 51456 124 12701 21756 9 25 0 66
1 32 0 23772 21332 7475600 0 0 52388 116 11749 19039 11 26 1 62
1 34 0 22460 21548 7476268 0 0 50240 224 12103 21552 10 26 1 63
4 32 0 24040 21568 7476792 0 0 47648 48 11124 19446 16 28 0 56
1 33 0 34148 21724 7466504 0 0 49376 176 11903 20681 10 25 0 65
2 30 0 26528 21812 7457576 0 0 47952 140 12054 20289 12 27 0 61
2 33 0 21932 21176 7362808 0 0 46452 152 10675 18546 20 44 0 35
1 32 0 83112 20880 7331484 0 0 31612 212 8110 13900 25 44 0 31
1 33 0 93380 21172 7408304 0 0 47200 8 11808 19313 16 29 1 54
0 33 0 44540 21620 7458652 0 0 49012 0 11591 19500 16 26 0 59
0 34 0 18808 21720 7484392 0 0 55748 72 12333 22001 14 26 0 61
2 33 0 49372 21792 7445084 0 0 51660 72 11300 19226 14 29 1 56
1 29 0 29428 21984 7474608 0 0 47012 100 11435 19647 12 26 1 61
0 34 0 48368 22056 7455700 0 0 51828 92 11687 17129 9 22 6 62
1 32 0 22856 22192 7481132 0 0 50236 56 12020 18495 7 23 10 60
1 32 0 31588 21856 7446720 0 0 42816 148 8546 14873 20 34 0 46
2 34 0 29140 21976 7475568 0 0 54300 80 12145 18508 9 24 0 67
0 31 0 56328 21916 7448088 0 0 54240 308 11338 19211 13 26 1 59
2 34 0 82520 22012 7422084 0 0 48188 2500 11085 17522 12 27 1 61
1 34 0 111192 22236 7395136 0 0 53464 148 10976 18427 14 27 2 57
0 34 0 138372 22236 7366372 0 0 54852 68 11892 20766 10 27 0 63
0 29 0 162528 21772 7343852 0 0 53272 152 11218 20261 13 30 0 57
2 32 0 182976 20816 7325088 0 0 54884 320 11411 20328 12 29 0 59
4 30 0 198720 21188 7310028 0 0 51832 64 11346 20141 12 28 0 60
2 32 0 224296 21412 7283760 0 0 57440 80 12182 21054 7 25 0 68
0 33 0 257388 21688 7230920 0 0 43468 76 8279 14542 17 36 1 46
0 32 0 283164 21948 7215088 0 0 59424 312 11404 18576 11 27 0 62
0 31 0 304912 22192 7202876 0 0 57208 332 11106 19843 12 30 1 58
5 33 0 324412 22476 7183688 0 0 54904 3372 12113 17509 10 28 0 62
0 31 0 342648 22764 7165584 0 0 52332 764 11325 18572 15 29 0 56
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 33 0 360796 22944 7147248 0 0 56184 0 11082 18153 13 28 0 59
4 32 0 389932 23160 7110380 0 0 52580 0 10303 16708 14 32 0 54
1 32 0 397500 23448 7110772 0 0 61384 704 11328 17227 11 27 0 62
0 34 0 417404 23636 7091068 0 0 64840 88 11162 17765 8 28 0 64
1 33 0 432128 23840 7075904 0 0 63924 0 11492 19978 9 29 0 63
4 30 0 445324 24096 7063000 0 0 56508 28 10265 18108 14 30 0 55
3 29 0 459872 24292 7048728 0 0 57696 28 10308 18048 15 29 0 56
4 35 0 479552 23604 7022828 0 0 61804 128 9398 15858 16 35 0 49
2 35 0 476912 23216 7033620 0 0 53160 4292 9876 15157 16 28 0 56
4 32 0 481204 23372 7027616 0 0 49936 5360 10565 14389 11 24 3 62
0 33 0 482680 23476 7026424 0 0 41828 7284 10270 13548 11 23 11 56
3 25 0 492028 23704 7018240 0 0 52380 28 11229 16098 9 24 2 64
1 29 0 493972 23852 7016800 0 0 43804 8276 10168 11859 11 20 16 53
0 15 0 499740 24152 7009836 0 0 59168 352 10580 14203 10 24 0 65
0 34 0 501368 24292 6998544 0 0 43720 16184 9313 8995 13 27 8 53
3 32 0 508156 24412 7002368 0 0 55380 16 10477 11689 9 22 1 68
1 32 0 512184 24100 6998600 0 0 59068 64 9879 13955 13 26 3 58
SLOWDOWN END:
2 33 0 514884 23976 6992604 0 0 65064 36 9387 13410 17 29 2 52
1 33 0 517184 23956 6991128 0 0 61928 4312 9447 13799 14 29 0 57
6 29 0 516760 24088 6993172 0 0 62488 48 9345 15244 15 31 0 54
2 32 0 519196 24164 6992144 0 0 58348 52 8889 14019 13 29 0 57
0 27 0 523164 24140 6987884 0 0 64236 48 9732 13547 10 27 0 63
0 32 0 526752 24280 6984548 0 0 69872 3628 9800 13413 10 27 0 63
2 30 0 531428 22372 6981968 0 0 70148 584 8541 13988 12 32 0 56
0 33 0 535408 21476 6978036 0 0 68468 28 8438 11965 13 30 0 58
4 32 0 536868 20264 6976120 0 0 61804 56 8579 12447 12 28 0 60
6 29 0 540132 19908 6976136 0 0 68492 0 8319 11009 11 29 4 56
3 30 0 541564 18328 6976900 0 0 62148 40 8110 12429 12 30 0 58
1 35 0 547948 17876 6943896 0 0 49368 4900 6412 9800 19 38 0 44
2 31 0 543504 18548 6972600 0 0 63916 8 8427 12450 16 30 0 54
0 31 0 544084 18840 6972920 0 0 66656 0 7090 10378 14 30 0 56
0 32 0 543832 18824 6972528 0 0 63120 28 7035 10658 18 31 0 51
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 30 0 544664 19044 6972648 0 0 66036 28 7686 12178 14 32 0 54
3 32 0 544600 19032 6971708 0 0 73316 1868 8394 10431 11 30 0 59
5 26 0 544344 19184 6972508 0 0 63624 20 6577 11294 16 33 0 50
4 30 0 543580 19324 6972368 0 0 74900 20 6303 10768 18 35 0 47
1 32 0 545632 18504 6971488 0 0 75020 0 6737 10601 14 35 0 52
2 32 0 546848 18580 6971276 0 0 67288 0 6572 9537 13 33 0 55
4 33 0 543584 18640 6969924 0 0 61624 9936 6660 9273 13 34 0 53
2 28 0 544736 18244 6970116 0 0 67992 2708 8948 8232 6 29 0 65
2 30 0 546020 18640 6970400 0 0 64668 16 6114 8186 19 33 0 48
2 30 0 546788 18636 6970812 0 0 69240 8 6064 9482 17 34 0 49
5 32 0 544292 18352 6969804 0 0 74688 16 5957 9517 18 34 0 47
4 30 0 547524 18420 6969464 0 0 69756 276 6042 8357 12 33 0 56
6 29 0 547172 18240 6970188 0 0 70352 4 5210 8836 20 35 0 45
3 34 0 548836 18276 6971172 0 0 69536 0 5356 8443 14 35 0 52
6 28 0 548132 18228 6970268 0 0 70156 0 5428 9326 16 35 0 49
14 32 0 550672 18260 6940452 0 0 64460 12 4352 8160 19 41 0 41
10 31 0 601424 17488 6858060 0 0 62296 296 4154 6203 22 47 0 31
2 31 0 578960 17024 6851724 0 0 49492 2608 4892 6413 25 41 0 35
3 30 0 636752 17156 6881784 0 0 65036 1656 5695 7811 14 36 0 51
4 29 0 572068 17856 6944868 0 0 62252 2412 4594 6801 19 35 0 47
4 27 0 549092 17948 6967216 0 0 74172 36 5121 7671 16 35 0 50
2 34 0 550948 17980 6965552 0 0 73168 100 4792 7743 15 35 0 50
1 30 0 551736 17880 6906628 0 0 64548 208 4575 7705 20 43 0 37
5 33 0 548824 17732 6871484 0 0 61272 2844 3969 6323 22 43 0 35
0 30 0 545432 17980 6891432 0 0 67960 24 4655 6606 22 38 0 41
2 33 0 577176 18008 6932612 0 0 70592 4 4881 7156 13 36 0 51
1 29 0 544784 18328 6965068 0 0 70356 16 4246 6638 15 34 0 51
6 28 0 542672 18312 6964812 0 0 61576 4288 4333 6364 13 35 0 52
1 32 0 545744 18208 6965052 0 0 67540 24 4870 7232 18 34 0 48
2 29 0 546384 18168 6965500 0 0 68172 4 4588 7651 14 36 0 51
4 28 0 545296 18236 6964344 0 0 67020 0 4207 6546 15 35 0 51
3 30 0 544720 18536 6962548 0 0 74644 12 4417 6375 17 34 0 49
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 35 0 538448 18744 6964720 0 0 78184 248 4145 5987 15 35 0 50
6 25 0 538960 18284 6966540 0 0 71532 24 3885 7001 17 35 0 48
4 29 0 543280 17416 6955780 0 0 60540 20 4040 6582 20 38 0 42
5 28 0 540528 17324 6962332 0 0 79480 12 4147 5974 13 36 0 52
3 33 0 538672 17536 6960964 0 0 67944 8 4066 5822 12 34 0 53
1 33 0 535220 17696 6963796 0 0 77724 1752 4156 6577 15 35 0 51
3 32 0 538548 17800 6964032 0 0 65540 0 3890 6294 17 36 0 47
0 29 0 538740 17688 6964280 0 0 60444 12 3860 6419 21 37 0 42
5 29 0 536244 16988 6964640 0 0 67836 8 4336 5963 15 35 0 51
6 28 0 534964 17092 6965148 0 0 68276 0 3934 5697 15 35 0 50
2 30 0 534132 17184 6963492 0 0 62960 412 3883 5663 16 35 0 49
3 26 0 534900 17368 6963036 0 0 61040 8 3685 7448 24 36 0 40
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/