I'm running 2.0.32pre6 on a i386DX40 with 8Megs. This box is doing some
logging over the serial port (1200 baud), and is basically idle all the
time. I log in from time to time to see if the box is OK - sometimes
through slogin. After an uptime of around 5 days and no problems
whatsoever, the kernel began to behave weird.
The kernel is rather basic - only SYN cookies and drop source routed
packets is selected. I don't use modules (support is compiled in).
Almost any command segfaults, but the logging (minicom capture) is still
up and running. There's no pattern in the working commands.
For example 'ls' is OK but 'ls -alsF' fails.
It seems (in the log) to be initiated by an slogin, but I'm not shure
that's the reason - only the trigger (see later). slogin has been running
flawlessly. I don't think the box is to blame: 2.0.30 had an uptime of 55
days on the box, before I upgraded.
>From Nov 21 09:20 every cron job has sent a mail with the complaint:
/bin/bash: can't load library 't'
Many of the oopses seems to occur in the ext2 code, but I think this is a
sideeffect. I have +20M free on the filesys and no problems. After closing
the minicom log and rebooting (even had to hit reset because 'shutdown -r
now' died in big pain) the entire log was intact and minicom doesen't
flush the log - only at fclose...
I think this is the sign of a memory leak. I remember to notice a *VERY*
high number at one of the buffer counts at the <shift><numlock> dump -
like +40000 ot so. I didn't pay attention to it because "I can see that
later in the log". However, it never made it to the log... @#$&^%$@ Should
have written that one down!!!
I you find this problem interesting and need some info, I can try to
provide it. I've still got the kernel running (after a reboot), and my
guess is 3 days until the first signs begin again... If I should pay
special attention to something, please say so.
Here goes my message file. A few things has been snipped: the sshd
messages, and login messages. It's not the entire file (flame precaution),
but the rest just show things getting worse (available on request). The
ksymoops output is also available on request (not included because I
think the problem is elsewhere)...
Sorry for the long post and thanks for some (otherwise) great kernels: I
also run 2.0.32pre6 on my 200MHz K6 with 48Megs and *NO* problems!!!
This box is not running all the time so max uptime is less than 2 days.
Anyway, an 8 Meg box is faster to find a memleak...
Well here (really) goes...
Nov 18 01:35:51 hobbes syslogd 1.3-3: restart.
Nov 18 01:35:52 hobbes kernel: klogd 1.3-3, log source = /proc/kmsg started.
Nov 18 01:35:55 hobbes kernel: Loaded 3103 symbols from /boot/System.map.
Nov 18 01:35:55 hobbes kernel: Symbols match kernel version 2.0.32.
Nov 18 01:35:55 hobbes kernel: No module symbols loaded.
Nov 18 01:35:55 hobbes kernel: Console: 16 point font, 400 scans
Nov 18 01:35:55 hobbes kernel: Console: colour VGA+ 80x25, 1 virtual console (max 63)
Nov 18 01:35:55 hobbes kernel: Calibrating delay loop.. ok - 7.94 BogoMIPS
Nov 18 01:35:55 hobbes kernel: Memory: 6856k/8192k available (576k kernel code, 384k reserved, 376k data)
Nov 18 01:35:55 hobbes kernel: Swansea University Computer Society NET3.035 for Linux 2.0
Nov 18 01:35:55 hobbes kernel: NET3: Unix domain sockets 0.13 for Linux NET3.035.
Nov 18 01:35:55 hobbes kernel: Swansea University Computer Society TCP/IP for NET3.034
Nov 18 01:35:55 hobbes kernel: IP Protocols: ICMP, UDP, TCP
Nov 18 01:35:55 hobbes kernel: Checking 386/387 coupling... Ok, fpu using old IRQ13 error reporting
Nov 18 01:35:55 hobbes kernel: Checking 'hlt' instruction... Ok.
Nov 18 01:35:55 hobbes kernel: Linux version 2.0.32 (root@viggo.ostenfeld.dk) (gcc version 2.7.2) #6 Mon Nov 17 06:05:33 MET 1997
Nov 18 01:35:55 hobbes kernel: Starting kswapd v 1.4.2.2
Nov 18 01:35:55 hobbes kernel: Serial driver version 4.13 with no serial options enabled
Nov 18 01:35:55 hobbes kernel: tty00 at 0x03f8 (irq = 4) is a 16450
Nov 18 01:35:55 hobbes kernel: tty01 at 0x02f8 (irq = 3) is a 16450
Nov 18 01:35:55 hobbes kernel: hda: st3120AT, 102MB w/32kB Cache, CHS=1024/12/17
Nov 18 01:35:55 hobbes kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Nov 18 01:35:55 hobbes kernel: Floppy drive(s): fd0 is 1.44M
Nov 18 01:35:55 hobbes kernel: FDC 0 is an 8272A
Nov 18 01:35:55 hobbes kernel: ne.c:v1.10 9/23/94 Donald Becker (becker@cesdis.gsfc.nasa.gov)
Nov 18 01:35:55 hobbes kernel: NE*000 ethercard probe at 0x300: 00 80 c8 2b 77 8f
Nov 18 01:35:55 hobbes kernel: eth0: NE2000 found at 0x300, using IRQ 10.
Nov 18 01:35:55 hobbes kernel: Partition check:
Nov 18 01:35:55 hobbes kernel: hda: hda1 hda2
Nov 18 01:35:55 hobbes kernel: VFS: Mounted root (ext2 filesystem) readonly.
Nov 18 01:35:55 hobbes kernel: Adding Swap: 12644k swap-space (priority -1)
Nov 23 00:15:59 hobbes sshd[23479]: log: Connection from 192.38.216.180 port 1023
Nov 23 00:16:00 hobbes kernel: Unable to handle kernel NULL pointer dereference at virtual address c0000180
Nov 23 00:16:00 hobbes kernel: current->tss.cr3 = 003e4000, 8r3 = 003e4000
Nov 23 00:16:00 hobbes kernel: *pde = 00102067
Nov 23 00:16:00 hobbes kernel: *pte = 00000000
Nov 23 00:16:00 hobbes kernel: Oops: 0000
Nov 23 00:16:00 hobbes kernel: CPU: 0
Nov 23 00:16:00 hobbes kernel: EIP: 0010:[__iget+454/544]
Nov 23 00:16:00 hobbes kernel: EFLAGS: 00010206
Nov 23 00:16:00 hobbes kernel: eax: 00000100 ebx: 0070e000 ecx: 00003490 edx: 001b0729
Nov 23 00:16:00 hobbes kernel: esi: 001afdd8 edi: 00000000 ebp: 001c1c34 esp: 00362ed8
Nov 23 00:16:00 hobbes kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Nov 23 00:16:00 hobbes kernel: Process sshd (pid: 23479, process nr: 21, stackpage=00362000)
Nov 23 00:16:00 hobbes kernel: Stack: 004d0200 001fc398 00000009 00342005 001fc398 00151425 001c1c34 00003490
Nov 23 00:16:00 hobbes kernel: 00000001 004d0200 00000001 00362f60 00000009 001fa520 00003490 0012975d
Nov 23 00:16:00 hobbes kernel: 004d0200 00342005 00000009 00362f60 00000009 00000001 000081a4 00362f94
Nov 23 00:16:00 hobbes kernel: Call Trace: [ext2_lookup+341/364] [lookup+221/244] [open_namei+516/1032] [do_open+87/284] [sys_open+57/112] [system_call+85/128]
Nov 23 00:16:00 hobbes kernel: Code: 66 ff 80 80 00 00 00 53 89 44 24 14 e8 55 fa ff ff 8b 44 24
Nov 23 00:16:00 hobbes kernel: Unable to handle kernel NULL pointer dereference at virtual address c000000e
Nov 23 00:16:00 hobbes kernel: current->tss.cr3 = 00101000, 8r3 = 00101000
Nov 23 00:16:00 hobbes kernel: *pde = 00102067
Nov 23 00:16:00 hobbes kernel: *pte = 00000000
Nov 23 00:16:00 hobbes kernel: Oops: 0000
Nov 23 00:16:00 hobbes kernel: CPU: 0
Nov 23 00:16:00 hobbes kernel: EIP: 0010:[close_fp+7/92]
Nov 23 00:16:00 hobbes kernel: EFLAGS: 00010202
Nov 23 00:16:00 hobbes kernel: eax: 00000000 ebx: 00000003 ecx: 001a14f8 edx: 00000000
Nov 23 00:16:00 hobbes kernel: esi: 00000000 edi: 00000001 ebp: 00737018 esp: 00362e14
Nov 23 00:16:00 hobbes kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Nov 23 00:16:00 hobbes kernel: Process sshd (pid: 23479, process nr: 21, stackpage=00362000)
Nov 23 00:16:00 hobbes kernel: Stack: 00000003 00000004 00000001 001158ad 00000000 0009002b 00000014 00363000
Nov 23 00:16:00 hobbes kernel: 00362e9c 0010ab90 0000000b 0018df8e 00102000 00000000 001f0000 00370c0c
Nov 23 00:16:00 hobbes kernel: 00000024 01800000 01000000 00370018 001102c6 0018e6e7 00362e9c 001f0000
Nov 23 00:16:01 hobbes kernel: Call Trace: [do_exit+273/488] [die_if_kernel+676/684] [<01800000>] [<01000000>] [do_page_fault+730/748] [do_page_fault+0/748] [error_code+64/80]
Nov 23 00:16:01 hobbes kernel: [ext2_bmap+268/580] [ext2_bmap+268/580] [__iget+454/544] [ext2_lookup+341/364] [lookup+221/244] [open_namei+516/1032] [do_open+87/284] [sys_open+57/112]
Nov 23 00:16:01 hobbes kernel: [system_call+85/128]
Nov 23 00:16:01 hobbes kernel: Code: 66 83 7e 0e 00 75 16 68 a0 03 19 00 e8 28 15 ff ff 31 c0 83
Nov 23 00:16:25 hobbes sshd[23483]: log: Connection from 192.38.216.180 port 1022
Nov 23 00:16:25 hobbes kernel: Unable to handle kernel NULL pointer dereference at virtual address c0000180
Nov 23 00:16:25 hobbes kernel: current->tss.cr3 = 0001f000, 8r3 = 0001f000
Nov 23 00:16:25 hobbes kernel: *pde = 00102067
Nov 23 00:16:25 hobbes kernel: *pte = 00000000
Nov 23 00:16:25 hobbes kernel: Oops: 0000
Nov 23 00:16:25 hobbes kernel: CPU: 0
Nov 23 00:16:25 hobbes kernel: EIP: 0010:[__iget+454/544]
Nov 23 00:16:25 hobbes kernel: EFLAGS: 00010206
Nov 23 00:16:25 hobbes kernel: eax: 00000100 ebx: 0070e000 ecx: 00003490 edx: 001b28c8
Nov 23 00:16:25 hobbes kernel: esi: 001afdd8 edi: 00000000 ebp: 001c1c34 esp: 005f8ed8
Nov 23 00:16:25 hobbes kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Nov 23 00:16:25 hobbes kernel: Process sshd (pid: 23483, process nr: 21, stackpage=005f8000)
Nov 23 00:16:25 hobbes kernel: Stack: 004d0200 00000001 00000009 0070b005 00151351 00151383 001c1c34 00003490
Nov 23 00:16:25 hobbes kernel: 00000001 004d0200 00000001 005f8f60 00000009 004d0200 00003490 0012975d
Nov 23 00:16:25 hobbes kernel: 004d0200 0070b005 00000009 005f8f60 00000009 00000001 000081a4 005f8f94
Nov 23 00:16:25 hobbes kernel: Call Trace: [ext2_lookup+129/364] [ext2_lookup+179/364] [lookup+221/244] [open_namei+516/1032] [do_open+87/284] [sys_open+57/112] [system_call+85/128]
Nov 23 00:16:25 hobbes kernel: Code: 66 ff 80 80 00 00 00 53 89 44 24 14 e8 55 fa ff ff 8b 44 24
Nov 23 00:16:25 hobbes kernel: Unable to handle kernel NULL pointer dereference at virtual address c000000e
Nov 23 00:16:25 hobbes kernel: current->tss.cr3 = 00101000, 8r3 = 00101000
Nov 23 00:16:25 hobbes kernel: *pde = 00102067
Nov 23 00:16:25 hobbes kernel: *pte = 00000000
Nov 23 00:16:25 hobbes kernel: Oops: 0000
Nov 23 00:16:25 hobbes kernel: CPU: 0
Nov 23 00:16:25 hobbes kernel: EIP: 0010:[close_fp+7/92]
Nov 23 00:16:25 hobbes kernel: EFLAGS: 00010202
Nov 23 00:16:25 hobbes kernel: eax: 00000000 ebx: 00000003 ecx: 001a14f8 edx: 00000000
Nov 23 00:16:25 hobbes kernel: esi: 00000000 edi: 00000001 ebp: 00737810 esp: 005f8e14
Nov 23 00:16:25 hobbes kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Nov 23 00:16:25 hobbes kernel: Process sshd (pid: 23483, process nr: 21, stackpage=005f8000)
Nov 23 00:16:25 hobbes kernel: Stack: 00000003 00000004 00000001 001158ad 00000000 0009002b 00000014 005f9000
Nov 23 00:16:25 hobbes kernel: 005f8e9c 0010ab90 0000000b 0018df8e 00102000 00000000 00000000 00375810
Nov 23 00:16:25 hobbes kernel: 00000024 01800000 01000000 00370018 001102c6 0018e6e7 005f8e9c 00000000
Nov 23 00:16:25 hobbes kernel: Call Trace: [do_exit+273/488] [die_if_kernel+676/684] [<01800000>] [<01000000>] [do_page_fault+730/748] [do_page_fault+0/748] [error_code+64/80]
Nov 23 00:16:25 hobbes kernel: [ext2_bmap+268/580] [ext2_bmap+268/580] [__iget+454/544] [ext2_lookup+129/364] [ext2_lookup+179/364] [lookup+221/244] [open_namei+516/1032] [do_open+87/284]
Nov 23 00:16:25 hobbes kernel: [sys_open+57/112] [system_call+85/128]
Nov 23 00:16:25 hobbes kernel: Code: 66 83 7e 0e 00 75 16 68 a0 03 19 00 e8 28 15 ff ff 31 c0 83
Nov 23 00:38:34 hobbes sshd[174]: log: Generating new 768 bit RSA key.
Nov 23 00:39:20 hobbes sshd[174]: log: RSA key generation complete.
Nov 23 01:02:02 hobbes syslogd 1.3-3: restart.
Nov 23 01:02:03 hobbes syslogd 1.3-3: restart.
Nov 23 01:02:04 hobbes syslogd 1.3-3: restart.
Nov 23 01:02:04 hobbes syslogd 1.3-3: restart.
Nov 23 02:02:19 hobbes kernel: Unable to handle kernel NULL pointer dereference at virtual address c0000100
Nov 23 02:02:20 hobbes kernel: current->tss.cr3 = 007f4000, 8r3 = 007f4000
Nov 23 02:02:20 hobbes kernel: *pde = 00102067
Nov 23 02:02:20 hobbes kernel: *pte = 00000000
Nov 23 02:02:20 hobbes kernel: Oops: 0000
Nov 23 02:02:20 hobbes kernel: CPU: 0
Nov 23 02:02:21 hobbes kernel: EIP: 0010:[__iget+60/544]
Nov 23 02:02:21 hobbes kernel: EFLAGS: 00010206
Nov 23 02:02:21 hobbes kernel: eax: 00000301 ebx: 00000100 ecx: 00000901 edx: 001b1839
Nov 23 02:02:21 hobbes kernel: esi: 001af150 edi: 00000000 ebp: 001c1c34 esp: 005a9e54
Nov 23 02:02:21 hobbes kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Nov 23 02:02:21 hobbes kernel: Process gawk (pid: 23837, process nr: 28, stackpage=005a9000)
Nov 23 02:02:21 hobbes kernel: Stack: 0032e000 00776798 00000006 0027bb90 00776798 00151425 001c1c34 00000901
Nov 23 02:02:21 hobbes kernel: 00000001 0032e000 00000001 005a9edc 00000006 00772730 00000901 0012975d
Nov 23 02:02:21 hobbes kernel: 0032e000 0027bb90 00000006 005a9edc 00000006 00000001 000081a4 005a9f60
Nov 23 02:02:21 hobbes kernel: Call Trace: [ext2_lookup+341/364] [lookup+221/244] [open_namei+516/1032] [ext2_follow_link+298/348] [lookup+221/244] [follow_link+97/104] [open_namei+549/1032]
Nov 23 02:02:21 hobbes kernel: [do_open+87/284] [sys_open+57/112] [system_call+85/128]
Nov 23 02:02:21 hobbes kernel: Code: 66 39 03 75 0d 8b 4c 24 1c 39 4b 04 0f 84 fa 00 00 00 8b 5b
[rest of messages snipped]
------------------------------------------------------------------------
Robin G. Theander E-mail: rgt@it.dtu.dk, robin@ostenfeld.dk
Raevehojvej 36, 810 Phone: +45 4587 1181 + 2810
DK-2800 Lyngby Mobil: +45 4036 4538
DENMARK Ham: OZ1RGT
------------------------------------------------------------------------
I haven't lost my mind - I've got it backed up on tape somewhere...