Re: System crash after "No irq handler for vector" linux 2.6.19

From: l . genoni
Date: Tue Jan 23 2007 - 05:37:01 EST

Next message: Alon Bar-Lev: "Re: [patch 03/26] Dynamic kernel command-line - arm"
Previous message: l . genoni: "Re: System crash after "No irq handler for vector" linux 2.6.19"
In reply to: l . genoni: "Re: System crash after "No irq handler for vector" linux 2.6.19"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Mon, 22 Jan 2007, Eric W. Biederman wrote:

"Luigi Genoni" <luigi.genoni@xxxxxxxxxxx> writes:

Hi,
this night a linux server 8 dual core CPU Optern 2600Mhz crashed just

after

giving this message

Jan 22 04:48:28 frey kernel: do_IRQ: 1.98 No irq handler for vector

Ok. This indicates that the hardware is doing something we didn't

expect.

We don't know which irq the hardware was trying to deliver when it
sent vector 0x98 to cpu 1.

I have no other logs, and I eventually lost the OOPS since I have no

net

console setled up.

If you had an oops it may have meant the above message was a secondary
symptom. Groan. If it stayed up long enough to give an OOPS then
there is a chance the above message appearing only once had nothing
to do with the actual crash.

How long had the system been up?

Sorry, my english is so bad, so I could not espress really what I wanted to
say.

I didn't get an OOPS. I could not see it on the console (nor logs). I do not
have netconsole

The system was up and running since 52 days.

As I said sistem is running linux 2.6.19 compiled with gcc 4.1.1 for

AMD

Opteron (attached see .config), no kernel preemption excepted the BKL
preemption. glibc 2.4.

System has 16 GB RAM and 8 dual core Opteron 2600Mhz.

I am running irqbalance 0.55.

any hints on what has happened?

Three guesses.

- A race triggered by irq migration (but I would expect more people to

yelling).
The code path where that message comes from is new in 2.6.19 so it may

not

have
had all of the bugs found yet :(
- A weird hardware or BIOS setup.
- A secondary symptom triggered by some other bug.

If this winds up being reproducible we should be able to track it down.
If not this may end up in the files of crap something bad happened that
we don't understand.

The one condition I know how to test for (if you are willing) is an
irq migration race. Simply by triggering irq migration much more often,
and thus increasing our chances of hitting a problem.

OK, will try tomorrow morning (I am at home right now, in Italy is night).

Stopping irqbalance and running something like:
for irq in 0 24 28 29 44 45 60 68 ; do
while :; do
for mask in 1 2 4 8 10 20 40 80 100 200 400 800 1000 2000

4000

8000 ; do
echo mask > /proc/irq/$irq/smp_affinity
sleep 1
done
done &
done

Should force every irq to migrate once a second, and removing the sleep

is even harsher, although we max at one irq migration by irq received.

If some variation of the above loop does not trigger the do_IRQ ??? No

irq

handler
for vector message chances are it isn't a race in irq migration.

If we can rule out the race scenario it will at least put us in the

right

direction
for guessing what went wrong with your box.

btw, the server is running tibco (bwengine, imse and adr3, all tibco software
is multithread) which use multicast and stress the net cards continuosly
sending out and receiving multicast and UDP packets talking with a remote
oracle DB on eth2 (gigabit ethernet). tibco continuously creates and deletes
small files (small often less than 1 KB, thousands per minute when it has a lot
of work to do) on three SAN luns (IBM DS8000) of 33 GB each one, accessed
trought 2 path fibre channels 2Gbit (linux dm multipath) with LVM2 volumes
(reiserFS, since gives us the best performance with all those small files, and
never gave us trobles).

In reality the system sees 21 luns on two paths, and those luns are also seen
by other 4 linux servers identical to this one clustered togheter with this
system using service guard (which is only userspace cluster software, no kernel
modules, eth0 and eth4 are HB and eth6 is the card on the public LAN ).
The Volume Group accessed by this server is reserved using LVM2 tags.

Tibco also does a normal I/O (about 2 MB/s) on a TCP NFS3 mounted volume using
eth2. Never had troubles with this NFS mount

Client nfs v3 (after 10 hour of uptime):
null getattr setattr lookup access readlink
0 0% 2770320 78% 122 0% 546401 15% 19672 0% 133 0%
read write create mkdir symlink mknod
40673 1% 3338 0% 58 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
32 0% 0 0% 0 0% 0 0% 144213 4% 552 0%
fsstat fsinfo pathconf commit
316 0% 872 0% 0 0% 16 0%

so network cards and SAN cards do work a lot. I include the boot messages,
maybe could help you to figure out how this server is configured.

what more... ah, usually the system during the sunday night is not working a
lot (not working at all), and it crashed a sunday night. Loocking the sar I see
strange statistics...

sar -I SUM INTR intr/s
04:40:01 sum 839.40
05:00:01 sum 62.10 <---
05:10:01 sum 245.63

sar -a
CPU %user %nice %system %iowait %idle
04:40:01 all 1.14 0.00 0.44 0.06 98.36
05:00:01 all 287.15 288.31 287.73 288.17 0.00 <---
05:10:01 all 0.00 0.00 0.00 0.01 99.98

but normally during the day:
10:30:01 all 88.41 0.00 0.81 0.05 10.73)

sar -W cswch/s
04:40:01 7738.69
05:00:01 478986467937.98 <---
05:10:01 194.36

(but normally during the day:
10:20:01 61264.85)

sar -b tps rtps wtps bread/s bwrtn/s
04:40:01 187.05 0.02 187.03 0.27 1777.81
05:00:01 106.24 111.47 106.29 108.40 59.48 <---
05:10:01 2.48 0.02 2.47 0.27 28.40

sar -c proc/s
04:40:01 0.34
05:00:01 478986468210.56 <--- this is absolutelly abnormal
05:10:01 0.03

That's all I can say about the race scenario. As you need more tests to be done, please tell me.

Eric

Thanx

Luigi GenoniInspecting /boot/System.map-2.6.19-smp
Loaded 22489 symbols from /boot/System.map-2.6.19-smp.
Symbols match kernel version 2.6.19.
No module symbols loaded - kernel modules not enabled.

klogd 1.4.1, log source = ksyslog started.
OS-provided physical RAM map:
<4> BIOS-e820: 0000000000000000 - 000000000009a000 (usable)
<4> BIOS-e820: 000000000009a000 - 00000000000a0000 (reserved)
<4> BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved)
<4> BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
<4> BIOS-e820: 00000000dfef0000 - 00000000dfefb000 (ACPI data)
<4> BIOS-e820: 00000000dfefb000 - 00000000dff00000 (ACPI NVS)
<4> BIOS-e820: 00000000dff00000 - 00000000e0000000 (reserved)
<4> BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
<4> BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
<4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
<4> BIOS-e820: 0000000100000000 - 0000000420000000 (usable)
<7>Entering add_active_range(0, 0, 154) 0 entries of 3200 used
<7>Entering add_active_range(0, 256, 917232) 1 entries of 3200 used
<7>Entering add_active_range(0, 1048576, 4325376) 2 entries of 3200 used
<4>end_pfn_map = 4325376
<6>DMI present.
<7>ACPI: RSDP (v002 PTLTD ) @ 0x00000000000f7480
<7>ACPI: XSDT (v001 PTLTD XSDT 0x06040000 LTP 0x00000000) @ 0x00000000dfef5719
<7>ACPI: FADT (v003 FSC D2003 0x06040000 PTEC 0x000f4240) @ 0x00000000dfef9754
<7>ACPI: SSDT (v001 PTLTD POWERNOW 0x06040000 LTP 0x00000001) @ 0x00000000dfef9848
<7>ACPI: SRAT (v001 AMD HAMMER 0x06040000 AMD 0x00000001) @ 0x00000000dfefaa0c
<7>ACPI: SSDT (v001 AMD-K8 AMD-ACPI 0x06040000 AMD 0x00000001) @ 0x00000000dfefaccc
<7>ACPI: SSDT (v001 AMD-K8 AMD-ACPI 0x06040000 AMD 0x00000001) @ 0x00000000dfefad69
<7>ACPI: MADT (v001 PTLTD APIC 0x06040000 LTP 0x00000000) @ 0x00000000dfefae06
<7>ACPI: BOOT (v001 PTLTD $SBFTBL$ 0x06040000 LTP 0x00000001) @ 0x00000000dfefaf88
<7>ACPI: SPCR (v001 PTLTD $UCRTBL$ 0x06040000 PTL 0x00000001) @ 0x00000000dfefafb0
<7>ACPI: DSDT (v001 FSC D2003 0x06040000 MSFT 0x02000002) @ 0x0000000000000000
<6>SRAT: PXM 0 -> APIC 0 -> Node 0
<6>SRAT: PXM 0 -> APIC 1 -> Node 0
<6>SRAT: PXM 1 -> APIC 2 -> Node 1
<6>SRAT: PXM 1 -> APIC 3 -> Node 1
<6>SRAT: PXM 2 -> APIC 4 -> Node 2
<6>SRAT: PXM 2 -> APIC 5 -> Node 2
<6>SRAT: PXM 3 -> APIC 6 -> Node 3
<6>SRAT: PXM 3 -> APIC 7 -> Node 3
<6>SRAT: PXM 4 -> APIC 8 -> Node 4
<6>SRAT: PXM 4 -> APIC 9 -> Node 4
<6>SRAT: PXM 5 -> APIC 10 -> Node 5
<6>SRAT: PXM 5 -> APIC 11 -> Node 5
<6>SRAT: PXM 6 -> APIC 12 -> Node 6
<6>SRAT: PXM 6 -> APIC 13 -> Node 6
<6>SRAT: PXM 7 -> APIC 14 -> Node 7
<6>SRAT: PXM 7 -> APIC 15 -> Node 7
<6>SRAT: Node 0 PXM 0 0-a0000
<7>Entering add_active_range(0, 0, 154) 0 entries of 3200 used
<6>SRAT: Node 0 PXM 0 0-80000000
<7>Entering add_active_range(0, 0, 154) 1 entries of 3200 used
<7>Entering add_active_range(0, 256, 524288) 1 entries of 3200 used
<6>SRAT: Node 1 PXM 1 80000000-e0000000
<7>Entering add_active_range(1, 524288, 917232) 2 entries of 3200 used
<6>SRAT: Node 1 PXM 1 80000000-120000000
<7>Entering add_active_range(1, 524288, 917232) 3 entries of 3200 used
<7>Entering add_active_range(1, 1048576, 1179648) 3 entries of 3200 used
<6>SRAT: Node 2 PXM 2 120000000-1a0000000
<7>Entering add_active_range(2, 1179648, 1703936) 4 entries of 3200 used
<6>SRAT: Node 3 PXM 3 1a0000000-220000000
<7>Entering add_active_range(3, 1703936, 2228224) 5 entries of 3200 used
<6>SRAT: Node 4 PXM 4 220000000-2a0000000
<7>Entering add_active_range(4, 2228224, 2752512) 6 entries of 3200 used
<6>SRAT: Node 5 PXM 5 2a0000000-320000000
<7>Entering add_active_range(5, 2752512, 3276800) 7 entries of 3200 used
<6>SRAT: Node 6 PXM 6 320000000-3a0000000
<7>Entering add_active_range(6, 3276800, 3801088) 8 entries of 3200 used
<6>SRAT: Node 7 PXM 7 3a0000000-420000000
<7>Entering add_active_range(7, 3801088, 4325376) 9 entries of 3200 used
<7>NUMA: Using 29 for the hash shift.
<6>Bootmem setup node 0 0000000000000000-0000000080000000
<6>Bootmem setup node 1 0000000080000000-0000000120000000
<6>Bootmem setup node 2 0000000120000000-00000001a0000000
<6>Bootmem setup node 3 00000001a0000000-0000000220000000
<6>Bootmem setup node 4 0000000220000000-00000002a0000000
<6>Bootmem setup node 5 00000002a0000000-0000000320000000
<6>Bootmem setup node 6 0000000320000000-00000003a0000000
<6>Bootmem setup node 7 00000003a0000000-0000000420000000
<4>Zone PFN ranges:
<4> DMA 0 -> 4096
<4> DMA32 4096 -> 1048576
<4> Normal 1048576 -> 4325376
<4>early_node_map[10] active PFN ranges
<4> 0: 0 -> 154
<4> 0: 256 -> 524288
<4> 1: 524288 -> 917232
<4> 1: 1048576 -> 1179648
<4> 2: 1179648 -> 1703936
<4> 3: 1703936 -> 2228224
<4> 4: 2228224 -> 2752512
<4> 5: 2752512 -> 3276800
<4> 6: 3276800 -> 3801088
<4> 7: 3801088 -> 4325376
<7>On node 0 totalpages: 524186
<7> DMA zone: 56 pages used for memmap
<7> DMA zone: 1070 pages reserved
<7> DMA zone: 2868 pages, LIFO batch:0
<7> DMA32 zone: 7112 pages used for memmap
<7> DMA32 zone: 513080 pages, LIFO batch:31
<7> Normal zone: 0 pages used for memmap
<7>On node 1 totalpages: 524016
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 7168 pages used for memmap
<7> DMA32 zone: 385776 pages, LIFO batch:31
<7> Normal zone: 1792 pages used for memmap
<7> Normal zone: 129280 pages, LIFO batch:31
<7>On node 2 totalpages: 524288
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 0 pages used for memmap
<7> Normal zone: 7168 pages used for memmap
<7> Normal zone: 517120 pages, LIFO batch:31
<7>On node 3 totalpages: 524288
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 0 pages used for memmap
<7> Normal zone: 7168 pages used for memmap
<7> Normal zone: 517120 pages, LIFO batch:31
<7>On node 4 totalpages: 524288
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 0 pages used for memmap
<7> Normal zone: 7168 pages used for memmap
<7> Normal zone: 517120 pages, LIFO batch:31
<7>On node 5 totalpages: 524288
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 0 pages used for memmap
<7> Normal zone: 7168 pages used for memmap
<7> Normal zone: 517120 pages, LIFO batch:31
<7>On node 6 totalpages: 524288
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 0 pages used for memmap
<7> Normal zone: 7168 pages used for memmap
<7> Normal zone: 517120 pages, LIFO batch:31
<7>On node 7 totalpages: 524288
<7> DMA zone: 0 pages used for memmap
<7> DMA32 zone: 0 pages used for memmap
<7> Normal zone: 7168 pages used for memmap
<7> Normal zone: 517120 pages, LIFO batch:31
<6>ACPI: PM-Timer IO Port: 0xc008
<7>ACPI: Local APIC address 0xfee00000
<6>ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
<6>Processor #0 (Bootup-CPU)
<6>ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
<6>Processor #1
<6>ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
<6>Processor #2
<6>ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
<6>Processor #3
<6>ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
<6>Processor #4
<6>ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
<6>Processor #5
<6>ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
<6>Processor #6
<6>ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
<6>Processor #7
<6>ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
<6>Processor #8
<6>ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled)
<6>Processor #9
<6>ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
<6>Processor #10
<6>ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled)
<6>Processor #11
<6>ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] enabled)
<6>Processor #12
<6>ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] enabled)
<6>Processor #13
<6>ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] enabled)
<6>Processor #14
<6>ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] enabled)
<6>Processor #15
<6>ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
<6>ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
<6>ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
<6>IOAPIC[0]: apic_id 1, address 0xfec00000, GSI 0-23
<6>ACPI: IOAPIC (id[0x02] address[0xe8000000] gsi_base[24])
<6>IOAPIC[1]: apic_id 2, address 0xe8000000, GSI 24-27
<6>ACPI: IOAPIC (id[0x03] address[0xe8001000] gsi_base[28])
<6>IOAPIC[2]: apic_id 3, address 0xe8001000, GSI 28-31
<6>ACPI: IOAPIC (id[0x06] address[0xf8010000] gsi_base[40])
<6>IOAPIC[3]: apic_id 6, address 0xf8010000, GSI 40-43
<6>ACPI: IOAPIC (id[0x07] address[0xf8011000] gsi_base[44])
<6>IOAPIC[4]: apic_id 7, address 0xf8011000, GSI 44-47
<6>ACPI: IOAPIC (id[0x0a] address[0xf8410000] gsi_base[56])
<6>IOAPIC[5]: apic_id 10, address 0xf8410000, GSI 56-59
<6>ACPI: IOAPIC (id[0x0b] address[0xf8411000] gsi_base[60])
<6>IOAPIC[6]: apic_id 11, address 0xf8411000, GSI 60-63
<6>ACPI: IOAPIC (id[0x0c] address[0xf8800000] gsi_base[64])
<6>IOAPIC[7]: apic_id 12, address 0xf8800000, GSI 64-67
<6>ACPI: IOAPIC (id[0x0d] address[0xf8801000] gsi_base[68])
<6>IOAPIC[8]: apic_id 13, address 0xf8801000, GSI 68-71
<6>ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
<7>ACPI: IRQ0 used by override.
<7>ACPI: IRQ2 used by override.
<7>ACPI: IRQ9 used by override.
<6>Setting APIC routing to physical flat
<6>Using ACPI (MADT) for SMP configuration information
<4>Nosave address range: 000000000009a000 - 00000000000a0000
<4>Nosave address range: 00000000000a0000 - 00000000000ce000
<4>Nosave address range: 00000000000ce000 - 0000000000100000
<4>Nosave address range: 00000000dfef0000 - 00000000dfefb000
<4>Nosave address range: 00000000dfefb000 - 00000000dff00000
<4>Nosave address range: 00000000dff00000 - 00000000e0000000
<4>Nosave address range: 00000000e0000000 - 00000000fec00000
<4>Nosave address range: 00000000fec00000 - 00000000fee00000
<4>Nosave address range: 00000000fee00000 - 00000000fee01000
<4>Nosave address range: 00000000fee01000 - 00000000fff80000
<4>Nosave address range: 00000000fff80000 - 0000000100000000
<6>Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
<6>PERCPU: Allocating 33664 bytes of per cpu data
<4>Built 8 zonelists. Total pages: 4133724
<5>Kernel command line: auto BOOT_IMAGE=Linux ro root=801
<4>Initializing CPU#0
<4>PID hash table entries: 4096 (order: 12, 32768 bytes)
<4>Console: colour VGA+ 80x25
<4>Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
<4>Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
<4>Checking aperture...
<4>CPU 0: aperture @ 0 size 32 MB
<4>No AGP bridge found
<4>Your BIOS doesn't leave a aperture memory hole
<4>Please enable the IOMMU option in the BIOS setup
<4>This costs you 64 MB of RAM
<4>Mapping aperture over 65536 KB of RAM @ 4000000
<4>Memory: 16443736k/17301504k available (2426k kernel code, 331984k reserved, 952k data, 316k init)
<4>Calibrating delay using timer specific routine.. 5197.21 BogoMIPS (lpj=25986093)
<6>Security Framework v1.0.0 initialized
<6>Capability LSM initialized
<4>Mount-cache hash table entries: 256
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 0/0 -> Node 0
<6>CPU: Physical Processor ID: 0
<6>CPU: Processor Core ID: 0
<6>Freeing SMP alternatives: 28k freed
<6>ACPI: Core revision 20060707
<6>Using local APIC timer interrupts.
<4>result 12483928
<6>Detected 12.483 MHz APIC timer.
<6>Booting processor 1/16 APIC 0x1
<4>Initializing CPU#1
<4>Calibrating delay using timer specific routine.. 5193.42 BogoMIPS (lpj=25967116)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 1/1 -> Node 0
<6>CPU: Physical Processor ID: 0
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 1: Syncing TSC to CPU 0.
<6>CPU 1: synchronized TSC with CPU 0 (last diff 2 cycles, maxerr 1685 cycles)
<6>Booting processor 2/16 APIC 0x2
<4>Initializing CPU#2
<4>Calibrating delay using timer specific routine.. 5193.42 BogoMIPS (lpj=25967140)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 2/2 -> Node 1
<6>CPU: Physical Processor ID: 1
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 2: Syncing TSC to CPU 0.
<6>CPU 2: synchronized TSC with CPU 0 (last diff 3 cycles, maxerr 1682 cycles)
<6>Booting processor 3/16 APIC 0x3
<4>Initializing CPU#3
<4>Calibrating delay using timer specific routine.. 5193.42 BogoMIPS (lpj=25967137)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 3/3 -> Node 1
<6>CPU: Physical Processor ID: 1
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 3: Syncing TSC to CPU 0.
<6>CPU 3: synchronized TSC with CPU 0 (last diff -1 cycles, maxerr 1687 cycles)
<6>Booting processor 4/16 APIC 0x4
<4>Initializing CPU#4
<4>Calibrating delay using timer specific routine.. 5193.42 BogoMIPS (lpj=25967144)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 4/4 -> Node 2
<6>CPU: Physical Processor ID: 2
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 4: Syncing TSC to CPU 0.
<6>CPU 4: synchronized TSC with CPU 0 (last diff -3 cycles, maxerr 1682 cycles)
<6>Booting processor 5/16 APIC 0x5
<4>Initializing CPU#5
<4>Calibrating delay using timer specific routine.. 5193.42 BogoMIPS (lpj=25967142)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 5/5 -> Node 2
<6>CPU: Physical Processor ID: 2
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 5: Syncing TSC to CPU 0.
<6>CPU 5: synchronized TSC with CPU 0 (last diff 12 cycles, maxerr 1685 cycles)
<6>Booting processor 6/16 APIC 0x6
<4>Initializing CPU#6
<4>Calibrating delay using timer specific routine.. 5193.41 BogoMIPS (lpj=25967061)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 6/6 -> Node 3
<6>CPU: Physical Processor ID: 3
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 6: Syncing TSC to CPU 0.
<6>CPU 6: synchronized TSC with CPU 0 (last diff -263 cycles, maxerr 1688 cycles)
<6>Booting processor 7/16 APIC 0x7
<4>Initializing CPU#7
<4>Calibrating delay using timer specific routine.. 5193.41 BogoMIPS (lpj=25967062)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 7/7 -> Node 3
<6>CPU: Physical Processor ID: 3
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 7: Syncing TSC to CPU 0.
<6>CPU 7: synchronized TSC with CPU 0 (last diff -262 cycles, maxerr 1686 cycles)
<6>Booting processor 8/16 APIC 0x8
<4>Initializing CPU#8
<4>Calibrating delay using timer specific routine.. 5193.41 BogoMIPS (lpj=25967055)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 8/8 -> Node 4
<6>CPU: Physical Processor ID: 4
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 8: Syncing TSC to CPU 0.
<6>CPU 8: synchronized TSC with CPU 0 (last diff -264 cycles, maxerr 1682 cycles)
<6>Booting processor 9/16 APIC 0x9
<4>Initializing CPU#9
<4>Calibrating delay using timer specific routine.. 5193.40 BogoMIPS (lpj=25967048)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 9/9 -> Node 4
<6>CPU: Physical Processor ID: 4
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 9: Syncing TSC to CPU 0.
<6>CPU 9: synchronized TSC with CPU 0 (last diff -251 cycles, maxerr 1662 cycles)
<6>Booting processor 10/16 APIC 0xa
<4>Initializing CPU#10
<4>Calibrating delay using timer specific routine.. 5193.40 BogoMIPS (lpj=25967008)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 10/a -> Node 5
<6>CPU: Physical Processor ID: 5
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 10: Syncing TSC to CPU 0.
<6>CPU 10: synchronized TSC with CPU 0 (last diff -269 cycles, maxerr 1708 cycles)
<6>Booting processor 11/16 APIC 0xb
<4>Initializing CPU#11
<4>Calibrating delay using timer specific routine.. 5193.39 BogoMIPS (lpj=25966996)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 11/b -> Node 5
<6>CPU: Physical Processor ID: 5
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 11: Syncing TSC to CPU 0.
<6>CPU 11: synchronized TSC with CPU 0 (last diff -269 cycles, maxerr 1706 cycles)
<6>Booting processor 12/16 APIC 0xc
<4>Initializing CPU#12
<4>Calibrating delay using timer specific routine.. 5193.40 BogoMIPS (lpj=25967008)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 12/c -> Node 6
<6>CPU: Physical Processor ID: 6
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 12: Syncing TSC to CPU 0.
<6>CPU 12: synchronized TSC with CPU 0 (last diff -269 cycles, maxerr 1697 cycles)
<6>Booting processor 13/16 APIC 0xd
<4>Initializing CPU#13
<4>Calibrating delay using timer specific routine.. 5193.40 BogoMIPS (lpj=25967000)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 13/d -> Node 6
<6>CPU: Physical Processor ID: 6
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 13: Syncing TSC to CPU 0.
<6>CPU 13: synchronized TSC with CPU 0 (last diff -272 cycles, maxerr 1700 cycles)
<6>Booting processor 14/16 APIC 0xe
<4>Initializing CPU#14
<4>Calibrating delay using timer specific routine.. 5193.47 BogoMIPS (lpj=25967367)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 14/e -> Node 7
<6>CPU: Physical Processor ID: 7
<6>CPU: Processor Core ID: 0
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 14: Syncing TSC to CPU 0.
<6>CPU 14: synchronized TSC with CPU 0 (last diff -284 cycles, maxerr 2789 cycles)
<6>Booting processor 15/16 APIC 0xf
<4>Initializing CPU#15
<4>Calibrating delay using timer specific routine.. 5193.47 BogoMIPS (lpj=25967386)
<6>CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
<6>CPU: L2 Cache: 1024K (64 bytes/line)
<6>CPU 15/f -> Node 7
<6>CPU: Physical Processor ID: 7
<6>CPU: Processor Core ID: 1
<4>Dual Core AMD Opteron(tm) Processor 885 stepping 02
<6>CPU 15: Syncing TSC to CPU 0.
<6>CPU 15: synchronized TSC with CPU 0 (last diff -297 cycles, maxerr 2802 cycles)
<6>Brought up 16 CPUs
<6>testing NMI watchdog ... OK.
<6>Disabling vsyscall due to use of PM timer
<6>time.c: Using 3.579545 MHz WALL PM GTOD PM timer.
<6>time.c: Detected 2596.653 MHz processor.
<4>migration_cost=644,695
<6>NET: Registered protocol family 16
<6>ACPI: bus type pci registered
<6>PCI: Using configuration type 1
<4>mtrr: your CPUs had inconsistent fixed MTRR settings
<6>mtrr: probably your BIOS does not setup all CPUs.
<6>mtrr: corrected configuration.
<6>ACPI: Interpreter enabled
<6>ACPI: Using IOAPIC for interrupt routing
<6>ACPI: PCI Root Bridge [PCI0] (0000:00)
<7>PCI: Probing PCI hardware (bus 00)
<7>Boot video device is 0000:01:05.0
<4>ACPI: PCI Interrupt Link [LNKA] (IRQs 3 *5 10 11)
<4>ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
<4>ACPI: PCI Interrupt Link [LNKC] (IRQs 3 5 10 11) *0, disabled.
<4>ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.TP2P._PRT]
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.G0PA._PRT]
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.G0PB._PRT]
<6>ACPI: PCI Root Bridge [PCI2] (0000:10)
<7>PCI: Probing PCI hardware (bus 10)
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI2.G2PA._PRT]
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI2.G2PB._PRT]
<6>ACPI: PCI Root Bridge [PCI5] (0000:28)
<7>PCI: Probing PCI hardware (bus 28)
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI5.G5PA._PRT]
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI5.G5PB._PRT]
<6>ACPI: PCI Root Bridge [PCI7] (0000:38)
<7>PCI: Probing PCI hardware (bus 38)
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI7.G7PA._PRT]
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI7.G7PB._PRT]
<6>Linux Plug and Play Support v0.97 (c) Adam Belay
<6>pnp: PnP ACPI init
<6>pnp: PnP ACPI: found 14 devices
<6>AMD768 RNG detected
<5>SCSI subsystem initialized
<6>PCI: Using ACPI for IRQ routing
<6>PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
<6>PCI-DMA: Disabling AGP.
<4>PCI-DMA: aperture base @ 4000000 size 65536 KB
<6>PCI-DMA: using GART IOMMU.
<6>PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
<6>pnp: 00:05: ioport range 0x4d0-0x4d1 has been reserved
<6>pnp: 00:05: ioport range 0x1100-0x117f has been reserved
<6>pnp: 00:05: ioport range 0x1180-0x11ff has been reserved
<6>pnp: 00:07: ioport range 0xca2-0xca5 has been reserved
<6>PCI: Bridge: 0000:00:06.0
<6> IO window: 2000-2fff
<6> MEM window: e8100000-e81fffff
<6> PREFETCH window: f0000000-f7ffffff
<6>PCI: Bridge: 0000:00:0a.0
<6> IO window: 3000-3fff
<6> MEM window: e8200000-e82fffff
<6> PREFETCH window: e2000000-e20fffff
<6>PCI: Bridge: 0000:00:0b.0
<6> IO window: 4000-4fff
<6> MEM window: e8300000-e83fffff
<6> PREFETCH window: e2100000-e21fffff
<6>PCI: Bridge: 0000:10:0c.0
<6> IO window: disabled.
<6> MEM window: disabled.
<6> PREFETCH window: disabled.
<6>PCI: Bridge: 0000:10:0d.0
<6> IO window: 5000-5fff
<6> MEM window: f8100000-f81fffff
<6> PREFETCH window: e2200000-e22fffff
<6>PCI: Bridge: 0000:28:05.0
<6> IO window: disabled.
<6> MEM window: disabled.
<6> PREFETCH window: disabled.
<6>PCI: Bridge: 0000:28:06.0
<6> IO window: disabled.
<6> MEM window: f8500000-f85fffff
<6> PREFETCH window: e2300000-e23fffff
<6>PCI: Bridge: 0000:38:07.0
<6> IO window: disabled.
<6> MEM window: disabled.
<6> PREFETCH window: disabled.
<6>PCI: Bridge: 0000:38:08.0
<6> IO window: disabled.
<6> MEM window: f8900000-f89fffff
<6> PREFETCH window: e2400000-e24fffff
<6>NET: Registered protocol family 2
<4>IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
<4>TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
<4>TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
<6>TCP: Hash tables configured (established 262144 bind 65536)
<6>TCP reno registered
<6>Simple Boot Flag at 0x37 set to 0x1
<3>microcode: CPU0 not a capable Intel processor
<3>microcode: CPU1 not a capable Intel processor
<3>microcode: CPU2 not a capable Intel processor
<3>microcode: CPU3 not a capable Intel processor
<3>microcode: CPU4 not a capable Intel processor
<3>microcode: CPU5 not a capable Intel processor
<3>microcode: CPU6 not a capable Intel processor
<3>microcode: CPU7 not a capable Intel processor
<3>microcode: CPU8 not a capable Intel processor
<3>microcode: CPU9 not a capable Intel processor
<3>microcode: CPU10 not a capable Intel processor
<3>microcode: CPU11 not a capable Intel processor
<3>microcode: CPU12 not a capable Intel processor
<3>microcode: CPU13 not a capable Intel processor
<3>microcode: CPU14 not a capable Intel processor
<3>microcode: CPU15 not a capable Intel processor
<6>IA-32 Microcode Update Driver: v1.14a <tigran@xxxxxxxxxxx>
<4>Total HugeTLB memory allocated, 0
<5>VFS: Disk quotas dquot_6.5.1
<4>Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
<6>io scheduler noop registered
<6>io scheduler cfq registered (default)
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:00:0a.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:00:0b.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:10:0c.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:10:0d.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:28:05.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:28:06.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:38:07.0 subordinate bus.
<4>PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:38:08.0 subordinate bus.
<6>ACPI: Power Button (FF) [PWRF]
<6>ACPI: Power Button (CM) [PWRB]
<4>Using specific hotkey driver
<6>Real Time Clock Driver v1.12ac
<6>Linux agpgart interface v0.101 (c) Dave Jones
<6>tg3.c:v3.69 (November 15, 2006)
<6>ACPI: PCI Interrupt 0000:03:04.0[A] -> GSI 28 (level, low) -> IRQ 28
<6>eth0: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c3:15
<6>eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth0: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:03:04.1[B] -> GSI 29 (level, low) -> IRQ 29
<6>eth1: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c3:16
<6>eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth1: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:12:04.0[A] -> GSI 44 (level, low) -> IRQ 44
<6>eth2: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c1:33
<6>eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth2: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:12:04.1[B] -> GSI 45 (level, low) -> IRQ 45
<6>eth3: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c1:34
<6>eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth3: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:2a:04.0[A] -> GSI 60 (level, low) -> IRQ 60
<6>eth4: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c2:48
<6>eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth4: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:2a:04.1[B] -> GSI 61 (level, low) -> IRQ 61
<6>eth5: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c2:49
<6>eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth5: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:3a:04.0[A] -> GSI 68 (level, low) -> IRQ 68
<6>eth6: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c2:5a
<6>eth6: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth6: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>ACPI: PCI Interrupt 0000:3a:04.1[B] -> GSI 69 (level, low) -> IRQ 69
<6>eth7: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:30:05:71:c2:5b
<6>eth7: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
<6>eth7: dma_rwctrl[769f0000] dma_mask[64-bit]
<6>Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
<6>ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
<6>megasas: 00.00.03.05 Mon Oct 02 11:21:32 PDT 2006
<6>Fusion MPT base driver 3.04.02
<6>Copyright (c) 1999-2005 LSI Logic Corporation
<6>Fusion MPT SPI Host driver 3.04.02
<6>Fusion MPT SAS Host driver 3.04.02
<6>ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 24 (level, low) -> IRQ 24
<6>mptbase: Initiating ioc0 bringup
<6>ioc0: SAS1068: Capabilities={Initiator}
<6>scsi0 : ioc0: LSISAS1068, FwRev=010a0100h, Ports=1, MaxQ=511, IRQ=24
<5>scsi 0:0:0:0: Direct-Access SEAGATE ST3146854SS 1503 PQ: 0 ANSI: 3
<5>scsi 0:0:0:0: Attached scsi generic sg0 type 0
<5>scsi 0:0:1:0: Direct-Access SEAGATE ST3146854SS 1503 PQ: 0 ANSI: 3
<5>scsi 0:0:1:0: Attached scsi generic sg1 type 0
<5>scsi 0:1:0:0: Direct-Access LSILOGIC Logical Volume 3000 PQ: 0 ANSI: 2
<5>SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
<5>sda: Write Protect is off
<7>sda: Mode Sense: 03 00 00 08
<5>SCSI device sda: drive cache: write through
<5>SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
<5>sda: Write Protect is off
<7>sda: Mode Sense: 03 00 00 08
<5>SCSI device sda: drive cache: write through
<6> sda: sda1 sda2 sda3
<5>sd 0:1:0:0: Attached scsi disk sda
<5>sd 0:1:0:0: Attached scsi generic sg2 type 0
<6>Fusion MPT misc device (ioctl) driver 3.04.02
<6>mptctl: Registered with Fusion MPT base driver
<6>mptctl: /dev/mptctl @ (major,minor=10,220)
<6>PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
<6>serio: i8042 KBD port at 0x60,0x64 irq 1
<6>serio: i8042 AUX port at 0x60,0x64 irq 12
<6>mice: PS/2 mouse device common for all mice
<6>input: PC Speaker as /class/input/input0
<6>input: AT Translated Set 2 keyboard as /class/input/input1
<6>device-mapper: ioctl: 4.10.0-ioctl (2006-09-14) initialised: dm-devel@xxxxxxxxxx
<6>EDAC MC: Ver: 2.0.1 Nov 30 2006
<6>TCP cubic registered
<6>NET: Registered protocol family 1
<6>NET: Registered protocol family 17
<4>drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
<6>input: ImExPS/2 Generic Explorer Mouse as /class/input/input2
<5>ReiserFS: sda1: found reiserfs format "3.6" with standard journal
<5>ReiserFS: sda1: using ordered data mode
<5>ReiserFS: sda1: journal params: device sda1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
<5>ReiserFS: sda1: checking transaction log (sda1)
<5>ReiserFS: sda1: replayed 1501 transactions in 15 seconds
<5>ReiserFS: sda1: Using r5 hash to sort names
<4>VFS: Mounted root (reiserfs filesystem) readonly.
<6>Freeing unused kernel memory: 316k freed
<5>ReiserFS: sda1: Removing [2 473 0x0 SD]..done
<5>ReiserFS: sda1: There were 1 uncompleted unlinks/truncates. Completed
<6>Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
<6>serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
<6>00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
<6>loop: loaded (max 8 devices)
<5>ReiserFS: dm-0: found reiserfs format "3.6" with standard journal
<5>ReiserFS: dm-0: using ordered data mode
<5>ReiserFS: dm-0: journal params: device dm-0, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
<5>ReiserFS: dm-0: checking transaction log (dm-0)
<5>ReiserFS: dm-0: replayed 114 transactions in 7 seconds
<5>ReiserFS: dm-0: Using r5 hash to sort names
<5>ReiserFS: dm-1: found reiserfs format "3.6" with standard journal
<5>ReiserFS: dm-1: using ordered data mode
<5>ReiserFS: dm-1: journal params: device dm-1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
<5>ReiserFS: dm-1: checking transaction log (dm-1)
<5>ReiserFS: dm-1: replayed 285 transactions in 15 seconds
<5>ReiserFS: dm-1: Using r5 hash to sort names
<5>ReiserFS: dm-2: found reiserfs format "3.6" with standard journal
<5>ReiserFS: dm-2: using ordered data mode
<5>ReiserFS: dm-2: journal params: device dm-2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
<5>ReiserFS: dm-2: checking transaction log (dm-2)
<5>ReiserFS: dm-2: replayed 870 transactions in 23 seconds
<5>ReiserFS: dm-2: Using r5 hash to sort names
<5>ReiserFS: dm-2: Removing [127 1895 0x0 SD]..done
<5>ReiserFS: dm-2: Removing [127 1893 0x0 SD]..done
<5>ReiserFS: dm-2: Removing [4 1889 0x0 SD]..done
<5>ReiserFS: dm-2: There were 3 uncompleted unlinks/truncates. Completed
<5>ReiserFS: dm-6: found reiserfs format "3.6" with standard journal
<5>ReiserFS: dm-6: using ordered data mode
<5>ReiserFS: dm-6: journal params: device dm-6, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
<5>ReiserFS: dm-6: checking transaction log (dm-6)
<5>ReiserFS: dm-6: replayed 152 transactions in 6 seconds
Kernel logging (ksyslog) stopped.
Kernel log daemon terminating.

Boot logging started on /dev/tty1(/dev/console) at Sat Jan 20 04:44:47 2007

Master Resource Control: previous runlevel: N, switching to runlevel: 3
<notice>startproc: execve (/usr/bin/dbus-daemon) [ /usr/bin/dbus-daemon --system ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/bin/dbus-daemon ]
<notice>startproc: execve (/sbin/resmgrd) [ /sbin/resmgrd ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/sbin/resmgrd ]
modprobe: FATAL: Module ac not found.

modprobe: FATAL: Module button not found.

modprobe: FATAL: Module fan not found.

modprobe: FATAL: Module processor not found.

modprobe: FATAL: Module thermal not found.

<notice>startproc: execve (/sbin/acpid) [ /sbin/acpid ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/sbin/acpid ]
[acpid]: starting up

[acpid]: 0 rules loaded

<notice>startproc: execve (/usr/sbin/irqbalance) [ /usr/sbin/irqbalance ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/irqbalance ]
/bin/mknod: `/dev/deadman': File exists
Created /dev/deadman c 10 62
raw driver installed.
Tivoli_lcf1: No such file or directory
acpid: loading ACPI modules ( battery ) failed
acpid: will skip unsupported modules
acpid: loading ACPI laptop modules ( ) done
Starting acpid done
FATAL: Module apparmor not found.
Loading AppArmor module failed
- could not start AppArmorfailed
cmcluster: No such file or directory
Initializing random number generatordone
Starting resource managerdone
Starting D-BUS daemondone
Starting irqbalance done
Starting HAL daemon<notice>startproc: execve (/usr/sbin/hald) [ /usr/sbin/hald --daemon=yes --retain-privileges ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/hald ]
done
Setting up network interfaces:
[acpid]: client connected from 3592[0:0]

[acpid]: 1 client rule loaded

lo
lo IP address: 127.0.0.1/8
done eth0 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
eth0 configuration: eth-id-00:30:05:71:c3:15
eth0 IP address: 10.201.25.204/22
Warning: Could not set up default route via interface
Command ip route replace to default via 10.201.11.1 returned:
. RTNETLINK answers: Network is unreachable
Configuration line: default 10.201.11.1 - -
This needs NOT to be AN ERROR if you set up multiple interfaces.
See man 5 routes how to avoid this warning.

done eth1 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
No configuration found for eth1
unused eth2 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
eth2 configuration: eth-id-00:30:05:71:c1:33
eth2 IP address: 10.201.21.204/22
Warning: Could not set up default route via interface
Command ip route replace to default via 10.201.11.1 returned:
. RTNETLINK answers: Network is unreachable
Configuration line: default 10.201.11.1 - -
This needs NOT to be AN ERROR if you set up multiple interfaces.
See man 5 routes how to avoid this warning.

done eth3 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
No configuration found for eth3
unused eth4 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
eth4 configuration: eth-id-00:30:05:71:c2:48
eth4 IP address: 10.201.10.204/24
Warning: Could not set up default route via interface
Command ip route replace to default via 10.201.11.1 returned:
. RTNETLINK answers: Network is unreachable
Configuration line: default 10.201.11.1 - -
This needs NOT to be AN ERROR if you set up multiple interfaces.
See man 5 routes how to avoid this warning.

done eth5 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
No configuration found for eth5
unused eth6 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
eth6 configuration: eth-id-00:30:05:71:c2:5a
eth6 IP address: 10.201.11.204/24
done eth7 device: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
No configuration found for eth7
unusedSetting up service network . . . . . . . . . . . . . . . .done
Starting syslog services<notice>startproc: execve (/sbin/syslogd) [ /sbin/syslogd -a /var/lib/dhcp/dev/log -a /var/lib/named/dev/log ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/sbin/syslogd ]
<notice>startproc: execve (/sbin/klogd) [ /sbin/klogd -c 1 -x -x ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/sbin/klogd ]
done
<notice>startproc: execve (/sbin/portmap) [ /sbin/portmap ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/sbin/portmap ]
Starting RPC portmap daemondone
Importing Net File System (NFS)unused
Checking/updating CPU microcode unused
Starting nfsboot (sm-notify) done
Starting ZENworks Management Daemondone
<notice>startproc: execve (/usr/sbin/sshd) [ /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/sshd ]
<notice>checkproc: /sbin/portmap 4298
<notice>startproc: execve (/usr/sbin/ypbind) [ /usr/sbin/ypbind ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/ypbind ]
Starting SSH daemondone
Jan 22 04:56:38 frey ntpdate[4397]: step time server 10.201.10.68 offset 173505.656746 sec

<notice>startproc: execve (/usr/sbin/ntpd) [ /usr/sbin/ntpd -p /var/lib/ntp/var/run/ntp/ntpd.pid -u ntp -i /var/lib/ntp ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/ntpd ]
Starting multipathdLoading keymap i386/qwerty/it.map.gz
doneStarting ypbind . .Jan 22 04:56:38 frey ntpd[4589]: ntpd 4.2.0a@xxxxxxxx Thu Jun 29 17:48:00 UTC 2006 (1)

<notice>startproc: execve (/usr/sbin/nscd) [ /usr/sbin/nscd ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/nscd ]
<notice>startproc: execve (/usr/sbin/atd) [ /usr/sbin/atd ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/atd ]
<notice>startproc: execve (/usr/sbin/famd) [ /usr/sbin/famd -t 4 -T 0 -L ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/famd ]
<notice>startproc: execve (/usr/sbin/sendmail) [ /usr/sbin/sendmail -L sendmail -Am -bd -q15m -om ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/sendmail ]
<notice>startproc: execve (/usr/sbin/sendmail) [ /usr/sbin/sendmail -L sendmail-client -Ac -qp1m ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/sendmail ]
<notice>startproc: execve (/usr/sbin/cron) [ /usr/sbin/cron ], [ CONSOLE=/dev/console SHELL=/bin/sh TERM=linux LC_ALL=POSIX INIT_VERSION=sysvinit-2.86 REDIRECT=/dev/tty1 AUTOBOOT=YES COLUMNS=80 PATH=/bin:/sbin:/usr/bin:/usr/sbin RUNLEVEL=3 PWD=/ SPLASHCFG= PREVLEVEL=N LINES=25 SHLVL=2 HOME=/ BOOT_IMAGE=Linux SPLASH=no _=/sbin/startproc DAEMON=/usr/sbin/cron ]
No NIS server and no -broadcast option specified.
Add a NIS server to the /etc/yp.conf configuration file,
or start ypbind with the -broadcast option.
done
Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd000000000000171d: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 65:160 1000 8:17

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd000000000000171e: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 65:176 1000 8:19

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd000000000000171f: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 65:192 1000 8:20

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001720: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 65:208 1000 8:22

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001721: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 65:224 1000 8:24

Jan 22 04:57:52 freyJan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001714: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:16

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001715: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:32

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001716: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:48

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001717: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:64

Jan 22 04:57:52 frey multipathd: 36005076303ffc4bd0000000000001718: load table [0 69206016 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 65:80

..dead
Try to get initial date and time via NTP from 10.201.10.68 10.201.10.69done
Starting network time protocol daemon (NTPD)done
Starting Name Service Cache Daemondone
Starting service at daemondone
Starting File Access Monitoring Daemondone
Loading compose table latin1.adddone
Start Unicode mode
doneLoading console font lat9w-16.psfu -m trivial G0:loadable
done . . . . . . . . . . . . . No NIS server founddone
Starting service automounter ("files nis" does not provide any mounts)skipped
Initializing SMTP port (sendmail)done
Starting CRON daemondone
Master Resource Control: runlevel 3 has been reached
Failed services in runlevel 3: Tivoli_lcf1 acpid cmcluster
Skipped services in runlevel 3: nfs microcode autofs
<notice>killproc: kill(3308,3)

Next message: Alon Bar-Lev: "Re: [patch 03/26] Dynamic kernel command-line - arm"
Previous message: l . genoni: "Re: System crash after "No irq handler for vector" linux 2.6.19"
In reply to: l . genoni: "Re: System crash after "No irq handler for vector" linux 2.6.19"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]