PROBLEM: kernel oops from tcp/ip stack

brynn@dilbert.mpl.ra.rockwell.com
Mon, 23 Aug 1999 09:27:38 -0500 (CDT)


PROBLEM: kernel oops from tcp/ip stack

When testing an ftp client against my linux wu-ftpd server 200,000 times over a
few days, the 2.2.10 SMP kernel generated this oops message.

modules, networking, smp, kernel

sh scripts/ver_linux gives this output
Linux dilbert.mpl.ra.rockwell.com 2.2.10 #7 SMP Tue Jul 27 10:56:18 CDT 1999
i686 unknown
Kernel modules 2.1.121
Gnu C egcs-2.91.66
Binutils 2.9.1.0.23
Linux C Library 2.1.1
Dynamic linker ldd (GNU libc) 2.1.1
Procps 2.0.2
Mount 2.9o
Net-tools 1.51
Kbd [option...]
Sh-utils 1.16
Modules Loaded loop nfsd lockd sunrpc tulip opl3 sb uart401 sound
soundlow soundcore

here is the oops run through ksymoops, I don't know why there are no symbols

Aug 16 18:00:34 dilbert kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Aug 16 18:00:34 dilbert kernel: current->tss.cr3 = 06335000, %cr3 = 06335000
Aug 16 18:00:34 dilbert kernel: *pde = 00000000
Aug 16 18:00:34 dilbert kernel: Oops: 0000
Aug 16 18:00:34 dilbert kernel: CPU: 1
Aug 16 18:00:34 dilbert kernel: EIP: 0010:[tcp_v4_rehash+227/344]
Aug 16 18:00:34 dilbert kernel: EFLAGS: 00010287
Aug 16 18:00:34 dilbert kernel: eax: 00000170 ebx: c7a7d400 ecx: c74d1b70 edx: 00000000
Aug 16 18:00:34 dilbert kernel: esi: 00000001 edi: c7a7d420 ebp: 00000000 esp: c74d7f6c
Aug 16 18:00:34 dilbert kernel: ds: 0018 es: 0018 ss: 0018
Aug 16 18:00:34 dilbert kernel: Process in.ftpd (pid: 21442, process nr: 106, stackpage=c74d7000)
Aug 16 18:00:34 dilbert kernel: Stack: 40184008 c01717ce c7a7d400 c33d56ec c74d6000 c014e83e c33d56ec 00000001
Aug 16 18:00:34 dilbert kernel: 00000003 00000002 c014f494 c014f4fb 00000007 00000001 c74d6000 00000007
Aug 16 18:00:34 dilbert kernel: 00000001 bfffecc0 00000000 00000000 00000000 c0109f74 00000004 bfffec90
Aug 16 18:00:34 dilbert kernel: Call Trace: [inet_listen+138/240] [sys_listen+70/112] [sys_socketcall+52/480] [sys_socketcall+155/480] [system_call+52/56]
Aug 16 18:00:34 dilbert kernel: Code: 66 39 0a 75 f8 8b 4a 08 85 c9 75 21 80 7b 2e 00 74 0f 8a 43

Code: 00000000 Before first symbol 00000000 <_IP>: <===
Code: 00000000 Before first symbol 0: 66 39 0a cmpw %cx,(%edx) <===
Code: 00000003 Before first symbol 3: 75 f8 jne fffffffd <END_OF_CODE+37772e51/????>
Code: 00000005 Before first symbol 5: 8b 4a 08 movl 0x8(%edx),%ecx
Code: 00000008 Before first symbol 8: 85 c9 testl %ecx,%ecx
Code: 0000000a Before first symbol a: 75 21 jne 0000002d Before first symbol
Code: 0000000c Before first symbol c: 80 7b 2e 00 cmpb $0x0,0x2e(%ebx)
Code: 00000010 Before first symbol 10: 74 0f je 00000021 Before first symbol
Code: 00000012 Before first symbol 12: 8a 43 00 movb 0x0(%ebx),%al

The environment is a 10BaseT network with dumb hubs, the ftp client is embedded
into a Windows CE product we are making using an SH4 cpu and aironet radio lan.
My linux box is a dual Pentium Pro 200Mhz running Redhat 6.0 with a 2.2.10 SMP
kernel, with 128MB of ECC EDO DRAM, the network card is a 4 port 10/100 Osicom
card (uses all DEC parts), eth0 and eth1 were up at the time of the crash
The windows CE device is running a big loop on it's custom ftp client code to
try and isolate bugs. It [windows CE tcp/ip stack] has crashed about a dozen
times compared to the one linux ftp server crash :).
Another interesting observation was that 15 seconds before the crash
the /var/log/xferlog file contained two streches of nulls in the middle of its
normal output (of about 3 transfers per second during this test)

The software
running was: wu-ftpd-2.4.2vr17-3

sh scripts/ver_linux:
Linux dilbert.mpl.ra.rockwell.com 2.2.10 #7 SMP Tue Jul 27 10:56:18 CDT 1999 i686 unknown
Kernel modules 2.1.121
Gnu C egcs-2.91.66
Binutils 2.9.1.0.23
Linux C Library 2.1.1
Dynamic linker ldd (GNU libc) 2.1.1
Procps 2.0.2
Mount 2.9o
Net-tools 1.51
Kbd [option...]
Sh-utils 1.16
Modules Loaded loop nfsd lockd sunrpc tulip opl3 sb uart401 sound soundlow soundcore

cat /proc/cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 1
model name : Pentium Pro
stepping : 9
cpu MHz : 199.312239
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
sep_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
bogomips : 199.07

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 1
model name : Pentium Pro
stepping : 9
cpu MHz : 199.312239
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
sep_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
bogomips : 198.66

cat /proc/modules
loop 7520 4 (autoclean)
nfsd 150744 8 (autoclean)
lockd 31208 1 (autoclean) [nfsd]
sunrpc 52612 1 (autoclean) [nfsd lockd]
tulip 25348 2 (autoclean)
opl3 11208 2
sb 33428 2
uart401 5968 2 [sb]
sound 57336 0 [opl3 sb uart401]
soundlow 224 0 [sound]
soundcore 2340 6 [sb sound]

cat /proc/pci

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: WDIGTL Model: ENTERPRISE Rev: 1.70
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: QUANTUM Model: XP39100W Rev: LYK8
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: QUANTUM Model: PD1225S Rev: 3142
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 04 Lun: 00
Vendor: MATSHITA Model: CD-R CW-7502 Rev: 4.10
Type: CD-ROM ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 05 Lun: 00
Vendor: IOMEGA Model: ZIP 100 Rev: J.03
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 06 Lun: 00
Vendor: PLEXTOR Model: CD-ROM PX-20TS Rev: 1.00
Type: CD-ROM ANSI SCSI revision: 02
brynn:dilbert ~/bug> cat /proc/pci
PCI devices found:
Bus 0, device 0, function 0:
Host bridge: Intel 82441FX Natoma (rev 2).
Medium devsel. Fast back-to-back capable. Master Capable. Latency=64.
Bus 0, device 7, function 0:
ISA bridge: Intel 82371SB PIIX3 ISA (rev 1).
Medium devsel. Fast back-to-back capable. Master Capable. No bursts.
Bus 0, device 7, function 1:
IDE interface: Intel 82371SB PIIX3 IDE (rev 0).
Medium devsel. Fast back-to-back capable. Master Capable. No bursts.
I/O at 0xecf0 [0xecf1].
Bus 0, device 15, function 0:
SCSI storage controller: Adaptec AIC-7881U (rev 0).
Medium devsel. Fast back-to-back capable. IRQ 19. Master Capable.
Latency=64. Min Gnt=8.Max Lat=8.
I/O at 0xf400 [0xf401].
Non-prefetchable 32 bit memory at 0xf0def000 [0xf0def000].
Bus 0, device 16, function 0:
VGA compatible controller: Number Nine Imagine 128v2 (rev 2).
Medium devsel. Fast back-to-back capable. IRQ 18.
Prefetchable 32 bit memory at 0xee000000 [0xee000008].
Prefetchable 32 bit memory at 0xee800000 [0xee800008].
Non-prefetchable 32 bit memory at 0xef800000 [0xef800000].
Non-prefetchable 32 bit memory at 0xf0000000 [0xf0000000].
Non-prefetchable 32 bit memory at 0xf0df0000 [0xf0df0000].
I/O at 0xf000 [0xf001].
Bus 0, device 17, function 0:
PCI bridge: DEC DC21052 (rev 1).
Medium devsel. Fast back-to-back capable. Master Capable. Latency=64.
Min Gnt=4.
Bus 1, device 4, function 0:
Ethernet controller: DEC DC21140 (rev 32).
Medium devsel. Fast back-to-back capable. IRQ 17. Master Capable.
Latency=165. Min Gnt=20.Max Lat=40.
I/O at 0xdc00 [0xdc01].
Non-prefetchable 32 bit memory at 0xef7ff800 [0xef7ff800].
Bus 1, device 5, function 0:
Ethernet controller: DEC DC21140 (rev 32).
Medium devsel. Fast back-to-back capable. IRQ 18. Master Capable.
Latency=165. Min Gnt=20.Max Lat=40.
I/O at 0xd880 [0xd881].
Non-prefetchable 32 bit memory at 0xef7ff400 [0xef7ff400].
Bus 1, device 6, function 0:
Ethernet controller: DEC DC21140 (rev 32).
Medium devsel. Fast back-to-back capable. IRQ 19. Master Capable.
Latency=165. Min Gnt=20.Max Lat=40.
I/O at 0xd800 [0xd801].
Non-prefetchable 32 bit memory at 0xef7ff000 [0xef7ff000].
Bus 1, device 7, function 0:
Ethernet controller: DEC DC21140 (rev 32).
Medium devsel. Fast back-to-back capable. IRQ 16. Master Capable.
Latency=165. Min Gnt=20.Max Lat=40.
I/O at 0xd480 [0xd481].
Non-prefetchable 32 bit memory at 0xef7fec00 [0xef7fec00].

cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: WDIGTL Model: ENTERPRISE Rev: 1.70
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: QUANTUM Model: XP39100W Rev: LYK8
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: QUANTUM Model: PD1225S Rev: 3142
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 04 Lun: 00
Vendor: MATSHITA Model: CD-R CW-7502 Rev: 4.10
Type: CD-ROM ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 05 Lun: 00
Vendor: IOMEGA Model: ZIP 100 Rev: J.03
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 06 Lun: 00
Vendor: PLEXTOR Model: CD-ROM PX-20TS Rev: 1.00
Type: CD-ROM ANSI SCSI revision: 02

http://www.visi.com/~brynn to see my triplets !

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/