In the early hours of Saturday (nobody was here) the following two oopses
were recorded (text + ksymoops translation):
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
general protection: 0000
CPU: 0
EIP: 0010:[<001223f0>]
EFLAGS: 00010086
eax: f000f84d ebx: 00000000 ecx: 00000000 edx: 00000206
esi: 0000044b edi: 00000000 ebp: bffffe94 esp: 00fc2f68
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process update (pid: 29, process nr: 7, stackpage=00fc2000)
Stack: 00000000 0000044b 00000000 00fc3018 00000000 00121e22 00000000 00fc3018
00000000 00000000 00125c50 00000000 00000000 00fc3018 00000000 00000000
00fc3018 00125d99 00fc3018 00000001 0010a692 00000001 00000000 00000000
Call Trace: [<00121e22>] [<00125c50>] [<00125d99>] [<0010a692>]
Using `./System.map' to map addresses to symbols.
>>EIP: 1223f0 <__wait_on_inode+38/bc>
Trace: 121e22 <sync_inodes+2a/54>
Trace: 125c50 <sync_old_buffers+14/128>
Trace: 125d99 <sys_bdflush+35/a4>
Trace: 10a692 <system_call+52/80>
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
and
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
general protection: 0000
CPU: 0
EIP: 0010:[<001223f0>]
EFLAGS: 00010086
eax: f000f84d ebx: 00000000 ecx: 00eb6044 edx: 00000212
esi: 0000044b edi: bfff0000 ebp: bffffdc8 esp: 00096f70
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process init (pid: 1, process nr: 1, stackpage=00096000)
Stack: 00000000 0000044b bfff0000 00097018 00000000 00121e22 00000000 00000000
bfffffce bfffffce 00122d02 00000000 00000000 00000000 00000000 00097018
00122d1f 00000000 0010a692 bffffddc 00000000 00000001 bfffffce bfffffce
Call Trace: [<00121e22>] [<00122d02>] [<00122d1f>] [<0010a692>]
Using `./System.map' to map addresses to symbols.
>>EIP: 1223f0 <__wait_on_inode+38/bc>
Trace: 121e22 <sync_inodes+2a/54>
Trace: 122d02 <fsync_dev+1a/30>
Trace: 122d1f <sys_sync+7/10>
Trace: 10a692 <system_call+52/80>
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
(note the lack of the "Code: " line. It just was not there in the
syslog output). The two happened in the same second (07:20:57). Then,
at 07:40:03 crontab started updatedb: I got:
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
Unable to handle kernel paging request at virtual address f03231a8
current->tss.cr3 = 00c3f000, yr3 = 00c3f000
*pde = 00000000
Oops: 0000
CPU: 0
EIP: 0010:[<00121fec>]
EFLAGS: 00010206
eax: 000002aa ebx: 30323130 ecx: 00001ff8 edx: 001e52a0
esi: 000003e8 edi: 00000000 ebp: 001e52a0 esp: 008e5ee0
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process find (pid: 5511, process nr: 34, stackpage=008e5000)
Stack: 00000000 001da590 00000000 00122231 00c965b8 0022a50c 00000004 00d1c000
0022a50c 0015c0df 001e52a0 00001ff8 00000001 00c965b8 008e5f74 008e5f74
Call Trace: [<00122231>] [<0015c0df>] [<0012a39e>] [<0012a5a4>] [<0012a63c>] [<00128871>] [<0010a692>]
Code: 66 83 7b 78 00 75 24 80 7b 7c 00 75 06 80 7b 7d 00 74 09 ba
Using `./System.map' to map addresses to symbols.
>>EIP: 121fec <get_empty_inode+44/150>
Trace: 122231 <__iget+61/1e8>
Trace: 15c0df <ext2_lookup+11f/138>
Trace: 12a39e <lookup+da/f4>
Trace: 12a5a4 <_namei+54/bc>
Trace: 12a63c <lnamei+30/48>
Trace: 128871 <sys_newlstat+29/58>
Trace: 10a692 <system_call+52/80>
Code: 121fec <get_empty_inode+44/150> cmpw $0x0,0x78(%ebx)
Code: 121ff1 <get_empty_inode+49/150> jne 122017 <get_empty_inode+6f/150>
Code: 121ff3 <get_empty_inode+4b/150> cmpb $0x0,0x7c(%ebx)
Code: 121ff7 <get_empty_inode+4f/150> jne 121fff <get_empty_inode+57/150>
Code: 121ff9 <get_empty_inode+51/150> cmpb $0x0,0x7d(%ebx)
Code: 121ffd <get_empty_inode+55/150> je 122008 <get_empty_inode+60/150>
Code: 121fff <get_empty_inode+57/150> movl $0x90909000,%edx
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
At 07:44:28 I got the firsty of a long series of oopses from nmbd
(samba daemon) that got restarted regularly because of requests from
other machines in the net:
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
Unable to handle kernel paging request at virtual address f03231a8
current->tss.cr3 = 00d5a000, ^^r3 = 00d5a000
*pde = 00000000
Oops: 0000
CPU: 0
EIP: 0010:[<00121fec>]
EFLAGS: 00010202
eax: 000002ad ebx: 30323130 ecx: 0065bf94 edx: 00adc1e8
esi: 000003e8 edi: 00000000 ebp: 000081a4 esp: 0065bec4
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process nmbd (pid: 2770, process nr: 12, stackpage=0065b000)
Stack: 00000000 0065bf34 00adc1e8 0015a073 00000000 0065bf34 00adc1e8 000081a4
001dbd10 0015c0a8 00adc1e8 00adc1e8 0015c004 00adc1e8 00adc1e8 00000242
Call Trace: [<0015a073>] [<0015c0a8>] [<0015c004>] [<0015c478>] [<0012a1d8>] [<0012a861>] [<00120d9c>]
[<00120ec2>] [<0010a692>]
Code: 66 83 7b 78 00 75 24 80 7b 7c 00 75 06 80 7b 7d 00 74 09 ba
Using `./System.map' to map addresses to symbols.
>>EIP: 121fec <get_empty_inode+44/150>
Trace: 15a073 <ext2_new_inode+13/648>
Trace: 15c0a8 <ext2_lookup+e8/138>
Trace: 15c004 <ext2_lookup+44/138>
Trace: 15c478 <ext2_create+38/160>
Trace: 12a1d8 <permission+20/94>
Trace: 12a861 <open_namei+1c5/3cc>
Trace: 120d9c <do_open+74/168>
Trace: 120ec2 <sys_open+32/58>
Trace: 10a692 <system_call+52/80>
Code: 121fec <get_empty_inode+44/150> cmpw $0x0,0x78(%ebx)
Code: 121ff1 <get_empty_inode+49/150> jne 122017 <get_empty_inode+6f/150>
Code: 121ff3 <get_empty_inode+4b/150> cmpb $0x0,0x7c(%ebx)
Code: 121ff7 <get_empty_inode+4f/150> jne 121fff <get_empty_inode+57/150>
Code: 121ff9 <get_empty_inode+51/150> cmpb $0x0,0x7d(%ebx)
Code: 121ffd <get_empty_inode+55/150> je 122008 <get_empty_inode+60/150>
Code: 121fff <get_empty_inode+57/150> movl $0x90909000,%edx
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
At 07:45:01 it was the turn of crond:
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
Unable to handle kernel paging request at virtual address f03231a8
current->tss.cr3 = 0082d000, ^^r3 = 0082d000
*pde = 00000000
Oops: 0000
CPU: 0
EIP: 0010:[<00121fec>]
EFLAGS: 00010202
eax: 000002ad ebx: 30323130 ecx: 00839f94 edx: 00e0c7a0
esi: 000003e8 edi: 00000000 ebp: 00008180 esp: 00839ec4
ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Process crond (pid: 185, process nr: 21, stackpage=00839000)
Stack: 00000000 00839f34 00e0c7a0 0015a073 00000000 00839f34 00e0c7a0 00008180
001da8c0 0015c0a8 00e0c7a0 00e0c7a0 0015c004 00e0c7a0 00e0c7a0 000006c2
Call Trace: [<0015a073>] [<0015c0a8>] [<0015c004>] [<0015c478>] [<0012a1d8>] [<0012a861>] [<00120d9c>]
[<00120ec2>] [<0010a692>]
Code: 66 83 7b 78 00 75 24 80 7b 7c 00 75 06 80 7b 7d 00 74 09 ba
Using `./System.map' to map addresses to symbols.
>>EIP: 121fec <get_empty_inode+44/150>
Trace: 15a073 <ext2_new_inode+13/648>
Trace: 15c0a8 <ext2_lookup+e8/138>
Trace: 15c004 <ext2_lookup+44/138>
Trace: 15c478 <ext2_create+38/160>
Trace: 12a1d8 <permission+20/94>
Trace: 12a861 <open_namei+1c5/3cc>
Trace: 120d9c <do_open+74/168>
Trace: 120ec2 <sys_open+32/58>
Trace: 10a692 <system_call+52/80>
Code: 121fec <get_empty_inode+44/150> cmpw $0x0,0x78(%ebx)
Code: 121ff1 <get_empty_inode+49/150> jne 122017 <get_empty_inode+6f/150>
Code: 121ff3 <get_empty_inode+4b/150> cmpb $0x0,0x7c(%ebx)
Code: 121ff7 <get_empty_inode+4f/150> jne 121fff <get_empty_inode+57/150>
Code: 121ff9 <get_empty_inode+51/150> cmpb $0x0,0x7d(%ebx)
Code: 121ffd <get_empty_inode+55/150> je 122008 <get_empty_inode+60/150>
Code: 121fff <get_empty_inode+57/150> movl $0x90909000,%edx
--8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<----8<--
(always at the same place). Similar stuff came from inetd at
07:55:46. Also from mgetty and httpd. Last message I got from gated at
14:23:38, always pointing to the same place, then probably syslogd
also went down. I connected from home on another box on the net, and I
found that I could not do ssh, telnet, rsh, or other stuff. But ping
was working OK, and the machine was GOING ON DOING ITS NETWORK ROUTING
BUSINESS!! So we went on receiving mail for all the time (the mail
server is on another machine) and I could talk to the internet as
normal!!
Anyway, when I got to the office today I tried to login & got an oop
that's similar to the ones above. So I had to use the Big Switch. When
rebooting I had to run fsck on my main disk by hand, because duplicate
inodes were found. E2fsck fixed everything and the box is now up and
running again :-)
If you need more info, please ask. It is the first time that I see
such stuff. I had to reboot the machine several times last week,
because of power downs. But it had been up since June, 27.
If you need more info, please ask.
Carlo
-- * Se la Strada e la sua Virtu' non fossero state messe da parte, * K * Carlo E. Prelz - fluido@marktest.pt che bisogno ci sarebbe * di parlare tanto di amore e di rettitudine? (Chuang-Tzu)