"mount: could not find filesystem" - aacraid? (was: Re: 2.6.26-git0: IDE oops during boot)
From: Bartlomiej Zolnierkiewicz
Date: Thu Feb 14 2008 - 06:48:50 EST
Hi,
On Thursday 14 February 2008, Kamalesh Babulal wrote:
> Bartlomiej Zolnierkiewicz wrote:
> > Hi,
> >
> > On Tuesday 12 February 2008, Kamalesh Babulal wrote:
> >> Bartlomiej Zolnierkiewicz wrote:
> >>> Hi,
> >>>
> >>> On Monday 11 February 2008, Kamalesh Babulal wrote:
> >>>> Nish Aravamudan wrote:
> >>>>> On 2/7/08, Bartlomiej Zolnierkiewicz <bzolnier@xxxxxxxxx> wrote:
> >>>>>> On Thursday 07 February 2008, Kamalesh Babulal wrote:
> >>>>>>> Bartlomiej Zolnierkiewicz wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> On Wednesday 06 February 2008, Pavel Machek wrote:
> >>>>>>>>> On Wed 2008-02-06 11:53:34, Pavel Machek wrote:
> >>>>>>>>>> Hi!
> >>>>>>>>>>
> >>>>>>>>>> Trying to boot 2.6.25-git0 (few days old), I get
> >>>>>>>>>>
> >>>>>>>>>> BUG: unable to handle kernel paging request at ffff..ffb0
> >>>>>>>>>> IP at init_irq+0x42e
> >>>>>>>> init_irq? hmm...
> >>>>>>>>
> >>>>>>>>>> Call trace:
> >>>>>>>>>> ide_device_add_all
> >>>>>>>> this comes from ide-generic
> >>>>>>>> (Generic IDE host driver)
> >>>>>>>>
> >>>>>>>>>> ide_generic_init
> >>>>>>>>>> kernel_init
> >>>>>>>>>> child_rip
> >>>>>>>>>> vgacon_cursor
> >>>>>>>>>> kernel_init
> >>>>>>>>>> child_rip
> >>>>>>>>>>
> >>>>>>>>>> Excerpt from config:
> >>>>>>>>>>
> >>>>>>>>>> CONFIG_IDE=y
> >>>>>>>>>> CONFIG_BLK_DEV_IDE=y
> >>>>>>>>> Disabling CONFIG_IDE made my machine boot, as it was using libata
> >>>>>>>>> anyway.
> >>>>>>>> Kamalesh/Pavel:
> >>>>>>>>
> >>>>>>>> Could you try latest git and see if the OOPS is still there?
> >>>>>>>>
> >>>>>>>> [ Yeah, I'm unable to reproduce it. :( ]
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Bart
> >>>>>>> Hi Bart,
> >>>>>>>
> >>>>>>> The panic is reproducible with the 2.6.24-git16 kernel, the call trace is
> >>>>>>> similar to the previous one
> >>>>>> Thanks, I again reviewed ide-probe.c changes but nothing seems wrong...
> >>>>>>
> >>>>>> Could you please bisect it down to the guilty commit?
> >>>>> Kamalesh, were you able to bisect this down? I just got hit by the
> >>>>> same panic on a 4-way x86_64, with 2.6.24-git22.
> >>>>>
> >>>>> Thanks,
> >>>>> Nish
> >>>> Hi Nish,
> >>>>
> >>>> I tried bisecting and the guilty patch seems to be
> >>>>
> >>>> 36501650ec45b1db308c3b51886044863be2d762 is first bad commit
> >>>> commit 36501650ec45b1db308c3b51886044863be2d762
> >>>> Author: Bartlomiej Zolnierkiewicz <bzolnier@xxxxxxxxx>
> >>>> Date: Fri Feb 1 23:09:31 2008 +0100
> >>>>
> >>>> ide: keep pointer to struct device instead of struct pci_dev in ide_hwif_t
> >>>>
> >>>>
> >>>> the gdb output, also points to the changes made by the guilty patch
> >>>>
> >>>> (gdb) p ide_device_add_all
> >>>> $1 = {int (u8 *, const struct ide_port_info *)} 0xffffffff804176ac <ide_device_add_all>
> >>>> (gdb) p/x 0xffffffff804176ac+0xb60
> >>>> $2 = 0xffffffff8041820c
> >>>> (gdb) l *0xffffffff8041820c
> >>>> 0xffffffff8041820c is in ide_device_add_all (drivers/ide/ide-probe.c:1249).
> >>>> 1244 goto out;
> >>>> 1245 }
> >>>> 1246
> >>>> 1247 sg_init_table(hwif->sg_table, hwif->sg_max_nents);
> >>>> 1248
> >>>> 1249 if (init_irq(hwif) == 0)
> >>>> 1250 goto done;
> >>>> 1251
> >>>> 1252 old_irq = hwif->irq;
> >>>> 1253 /*
> >>>> (gdb)
> >>>>
> >>>>
> >>>> (gdb) p init_irq
> >>>> $1 = {int (ide_hwif_t *)} 0xffffffff8041721f <init_irq>
> >>>> (gdb) p/x 0xffffffff8041721f+0x1a4
> >>>> $2 = 0xffffffff804173c3
> >>>> (gdb) l *0xffffffff804173c3
> >>>> 0xffffffff804173c3 is in init_irq (include/asm/pci.h:101).
> >>>> 96 /* Returns the node based on pci bus */
> >>>> 97 static inline int __pcibus_to_node(struct pci_bus *bus)
> >>>> 98 {
> >>>> 99 struct pci_sysdata *sd = bus->sysdata;
> >>>> 100
> >>>> 101 return sd->node;
> >>>> 102 }
> >>>> 103
> >>>> 104 static inline cpumask_t __pcibus_to_cpumask(struct pci_bus *bus)
> >>>> 105 {
> >>>> (gdb)
> >>> Thanks for the detailed analysis and sorry for the bug.
> >>>
> >>> I think that this may has been just fixed by Andi's recent hwif_to_node()
> >>> fix (patch below, it is in Linus' tree already), could please verify this?
> >>>
> >>> commit 1f07e988290fc45932f5028c9e2a862c37a57336
> >>> Author: Andi Kleen <andi@xxxxxxxxxxxxxx>
> >>> Date: Mon Feb 11 01:35:20 2008 +0100
> >>>
> >>> Prevent IDE boot ops on NUMA system
> >>>
> >>> Without this patch a Opteron test system here oopses at boot with
> >>> current git.
> >>>
> >>> Calling to_pci_dev() on a NULL pointer gives a negative value so the
> >>> following NULL pointer check never triggers and then an illegal address
> >>> is referenced. Check the unadjusted original device pointer for NULL
> >>> instead.
> >>>
> >>> Signed-off-by: Andi Kleen <ak@xxxxxxx>
> >>> Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> >>>
> >>> diff --git a/include/linux/ide.h b/include/linux/ide.h
> >>> index 23fad89..a3b69c1 100644
> >>> --- a/include/linux/ide.h
> >>> +++ b/include/linux/ide.h
> >>> @@ -1295,7 +1295,7 @@ static inline void ide_dump_identify(u8 *id)
> >>> static inline int hwif_to_node(ide_hwif_t *hwif)
> >>> {
> >>> struct pci_dev *dev = to_pci_dev(hwif->dev);
> >>> - return dev ? pcibus_to_node(dev->bus) : -1;
> >>> + return hwif->dev ? pcibus_to_node(dev->bus) : -1;
> >>> }
> >>>
> >>> static inline ide_drive_t *ide_get_paired_drive(ide_drive_t *drive)
> >> Hi Bart,
> >> Thanks !! the patch solves the kernel panic but when after applying the patch,kernel is not
> >> able to mount the filesystem and panics, am i not sure what is likely causing the panic.
> >
> > Is
> >
> > - the commit 36501650ec45b1db308c3b51886044863be2d762 with Andi's fix applied
> >
> > or
> >
> > - the commit f6fb786d6dcdd7d730e4fba620b071796f487e1b
> > (the one before commit 36501650ec45b1db308c3b51886044863be2d762)
> >
> > working for you?
>
> No, the commit before the commit 36501650ec45b1db308c3b51886044863be2d762 did not either work, i
> get the same kernel panic.
>
> >
> >> Creating root device.
> >> Mounting root filesystem.
> >> mount: could not find filesystem
> >> Kernel panic - not syncing: Attempted to kill init!
> >
> > Is IDE actually used for the boot device?
> >
> > [ Please send a dmesg output from the working system. ]
Hmm, it is not (from dmesg):
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
hda: HL-DT-STCD-RW/DVD DRIVE GCC-4244N, ATAPI CD/DVD-ROM drive
Probing IDE interface ide1...
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: ATAPI 24X DVD-ROM CD-R/RW drive, 2048kB Cache
Uniform CD-ROM driver Revision: 3.20
[...]
Adaptec aacraid driver 1.1-5[2449]-ms
ACPI: PCI Interrupt 0000:01:02.0[A] -> GSI 25 (level, low) -> IRQ 25
AAC0: kernel 5.2-0[11835] Jan 9 2007
AAC0: monitor 5.2-0[11835]
AAC0: bios 5.2-0[11835]
AAC0: serial 1625D1
AAC0: 64bit support enabled.
AAC0: 64 Bit DAC enabled
scsi0 : ServeRAID
scsi 0:0:0:0: Direct-Access IBM x366 V1.0 PQ: 0 ANSI: 2
scsi 0:1:0:0: Direct-Access IBM-ESXS ST973401SS B519 PQ: 0 ANSI: 5
scsi 0:1:1:0: Direct-Access IBM-ESXS ST973401SS B519 PQ: 0 ANSI: 5
scsi 0:1:2:0: Direct-Access IBM-ESXS ST973401SS B519 PQ: 0 ANSI: 5
scsi 0:3:0:0: Enclosure IBM SAS SES-2 DEVICE 0.09 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 429459456 512-byte hardware sectors (219883 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
sd 0:0:0:0: [sda] 429459456 512-byte hardware sectors (219883 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
sda: sda1 sda2 sda3 sda4 < sda5 sda6 >
sd 0:0:0:0: [sda] Attached SCSI removable disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 0:1:0:0: Attached scsi generic sg1 type 0
scsi 0:1:1:0: Attached scsi generic sg2 type 0
scsi 0:1:2:0: Attached scsi generic sg3 type 0
scsi 0:3:0:0: Attached scsi generic sg4 type 13
[...]
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
[...]
EXT3 FS on sda1, internal journal
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
I worry that another git-bisect session will be needed unless SCSI
developers are already aware of the problem source.
Thanks,
Bart
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/