Re: huge load average follow up

Madhusudana Rao (madhur@sasi.ernet.in)
Tue, 29 Apr 1997 12:20:19 +0500 (GMT+0500)


Hi,

On Mon, 28 Apr 1997, Andi Gutmans wrote:

andi -> Well this seems to be happening to people but noone knows what it is.
andi -> Is this ever going to be fixed Linus? I'm keeping my system up in case
andi -> someone wants debugging information. Load average is now 34. The system is
andi -> very usable with 20 people logged in etc...

I too got this atleast twice on a very standard config 2.0.30 system. I
see a lot of sendmail processes hung in "D" status (uninterruptible
sleep) and none of these could be killed. I see a load average of 22+ and
the system is very much usable. I could ofcourse "killall sendmail" and
the load average came down to our normal 0.5 or so.

The following are the logs if they help. If anybody has any clue and if I
need to do something that would be of help please let me know. I am
trying to see if I could reproduce this scenario.

Madhu

PS : tcpdump did not give me any clues.

========================== netstat -nato =========================
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (State) User
tcp 0 0 0.0.0.0:7 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:9 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:13 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:19 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:37 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:21 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:23 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:514 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:513 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:79 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 0 0 0.0.0.0:25 0.0.0.0:0 CLOSE root on (0.87/0)
tcp 0 0 0.0.0.0:53 0.0.0.0:0 LISTEN root off (0.00/0)
tcp 1 0 164.164.56.2:25 202.248.97.131:1816 CLOSE root on (0.15/1)
tcp 0 0 164.164.56.2:25 192.35.39.24:60616 CLOSE root on (0.77/1)
tcp 0 0 164.164.56.2:25 204.143.127.2:2489 CLOSE root on (0.37/1)
tcp 0 0 164.164.56.2:25 15.253.72.10:3708 CLOSE root on (1.36/0)
tcp 0 0 164.164.56.2:25 206.86.247.122:1790 CLOSE root on (0.57/1)
tcp 1 0 164.164.56.2:25 130.131.48.11:3141 CLOSE root on (1.60/1)
tcp 1 0 164.164.56.2:25 158.140.2.1:1921 CLOSE root on (2.14/1)
tcp 0 0 164.164.56.2:25 128.192.232.10:1278 CLOSE root on (0.50/1)
tcp 1 0 164.164.56.2:25 192.68.44.33:3113 CLOSE root on (0.76/5)
tcp 1 0 164.164.56.2:25 198.87.19.70:12637 CLOSE root on (0.39/2)
tcp 1 0 164.164.56.70:25 164.164.56.14:1562 CLOSE root on (0.96/1)
tcp 0 0 164.164.56.2:25 192.96.21.252:1307 ESTABLISHED root on (200.58/0)
tcp 1 0 164.164.56.70:25 164.164.56.20:3944 CLOSE root on (1.88/1)
tcp 1 0 164.164.56.70:25 164.164.56.20:3946 CLOSE root on (1.21/1)
tcp 1 0 164.164.56.70:25 164.164.56.14:1563 CLOSE root on (1.04/1)
tcp 1 0 164.164.56.2:25 137.65.40.4:2726 CLOSE root on (1.94/3)
tcp 1 0 164.164.56.2:25 198.87.19.70:16840 CLOSE root on (0.76/1)
tcp 1 0 164.164.56.2:25 139.23.36.11:30474 CLOSE root on (1.00/1)
tcp 1 0 164.164.56.2:25 164.164.128.17:54837 CLOSE root on (2.31/1)
tcp 1 0 164.164.56.70:25 164.164.56.20:3948 CLOSE root on (1.01/1)
tcp 1 0 164.164.56.70:25 164.164.56.20:3950 CLOSE root on (1.13/1)
tcp 1 0 164.164.56.70:25 164.164.56.20:3952 CLOSE root on (2.26/1)
tcp 1 0 164.164.56.70:25 164.164.56.20:3954 CLOSE root on (2.33/1)
tcp 0 2 164.164.56.70:513 202.21.147.100:1023 ESTABLISHED root on (0.28/0)
tcp 0 0 164.164.56.70:513 164.164.56.13:1022 ESTABLISHED root on (274.01/0)
tcp 0 205 164.164.56.70:513 164.164.56.13:1020 ESTABLISHED root on (0.36/0)
tcp 0 0 164.164.56.70:19877 202.21.147.10:23 ESTABLISHED root off (0.00/0)

The following are the /proc/locks file for the corresponding sendmail
processes.

============================= /proc/locks ==============================
1: POSIX ADVISORY WRITE 10769 08:03:77087 0 2147483647 00cb9718 00000000 008d9d18 00000000 00000000
1:
2: POSIX ADVISORY WRITE 10767 08:03:77083 0 2147483647 008d9d18 00cb9718 00724b58 00000000 00000000
2:
3: POSIX ADVISORY WRITE 10765 08:03:77081 0 2147483647 00724b58 008d9d18 00def598 00000000 00000000
3:
4: POSIX ADVISORY WRITE 10764 08:03:77076 0 2147483647 00def598 00724b58 007eb198 00000000 00000000
4:
5: POSIX ADVISORY WRITE 10762 08:03:77073 0 2147483647 007eb198 00def598 00af0698 00000000 00000000
5:
6: POSIX ADVISORY WRITE 10759 08:03:77070 0 2147483647 00af0698 007eb198 007d6b98 00000000 00000000
6:
7: POSIX ADVISORY WRITE 10758 08:03:77068 0 2147483647 007d6b98 00af0698 00d5d958 00000000 00000000
7:
8: POSIX ADVISORY WRITE 10757 08:03:77066 0 2147483647 00d5d958 007d6b98 00d5da58 00000000 00000000
8:
9: POSIX ADVISORY WRITE 10753 08:03:77062 0 2147483647 00d5da58 00d5d958 00d5d658 00000000 00000000
9:
10: POSIX ADVISORY WRITE 10751 08:03:77060 0 2147483647 00d5d658 00d5da58 00cafad8 00000000 00000000
10:
11: POSIX ADVISORY WRITE 10750 08:03:77056 0 2147483647 00cafad8 00d5d658 0097a498 00000000 00000000
11:
12: POSIX ADVISORY WRITE 10748 08:03:77053 0 2147483647 0097a498 00cafad8 00439198 00000000 00000000
12:
13: POSIX ADVISORY WRITE 10745 08:03:77050 0 2147483647 00439198 0097a498 004c6198 00000000 00000000
13:
14: POSIX ADVISORY WRITE 10744 08:03:77042 0 2147483647 004c6198 00439198 0055be98 00000000 00000000
14:
15: POSIX ADVISORY WRITE 10742 08:03:77038 0 2147483647 0055be98 004c6198 0044c718 00000000 00000000
15:
16: POSIX ADVISORY WRITE 10729 08:03:76941 0 2147483647 0044c718 0055be98 007d67d8 00000000 00000000
16:
17: POSIX ADVISORY WRITE 10722 08:03:76931 0 2147483647 007d67d8 0044c718 00d5d258 00000000 00000000
17:
18: POSIX ADVISORY WRITE 10721 08:03:76915 0 2147483647 00d5d258 007d67d8 00c0a258 00000000 00000000
18:
19: POSIX ADVISORY WRITE 10700 08:03:76939 0 2147483647 00c0a258 00d5d258 0083ce98 00000000 00000000
19:
20: POSIX ADVISORY WRITE 10700 08:03:76934 0 2147483647 0083ce98 00c0a258 00275e58 00000000 00000000
20:
21: POSIX ADVISORY WRITE 10695 08:03:77049 0 2147483647 00275e58 0083ce98 0032bf18 00000000 00000000
21:
22: POSIX ADVISORY WRITE 10695 08:03:77043 0 2147483647 0032bf18 00275e58 00def5d8 00000000 00000000
22:
23: POSIX ADVISORY WRITE 10655 08:03:77030 0 2147483647 00def5d8 0032bf18 00c0a658 00000000 00000000
23:
24: POSIX ADVISORY WRITE 10655 08:03:76933 0 2147483647 00c0a658 00def5d8 007d60d8 00000000 00000000
24:
25: POSIX ADVISORY WRITE 10618 08:03:76940 0 2147483647 007d60d8 00c0a658 0055b658 00000000 00000000
25:
26: POSIX ADVISORY WRITE 10490 08:03:76943 0 2147483647 0055b658 007d60d8 00000000 00000000 00000000
26:
=======================================================================