Re: BUG: unable to handle kernel paging request at ffffe8ff7fc00001

[Kyle Sanderson]

Looks like massive corruption, so the oops above probably isn't
anywhere near the cause. NFS was bouncing a bit, dmesg below...

I've downgraded back to 3.14.56, will see if it continues.

[523529.636418] nfs: server ftpback-bhs1-9.ip-198-100-151.net OK
[532541.586437] ntfs: driver 2.1.32 [Flags: R/O MODULE].
[532541.657558] REISERFS warning (device sda4): sh-2006
read_super_block: bread failed (dev sda4, block 8, size 1024)
[532541.657621] REISERFS warning (device sda4): sh-2006
read_super_block: bread failed (dev sda4, block 64, size 1024)
[532541.657682] REISERFS warning (device sda4): sh-2021
reiserfs_fill_super: can not find reiserfs on sda4
[532541.658379] EXT3-fs (sda4): error: unable to read superblock
[532541.659020] EXT2-fs (sda4): error: unable to read superblock
[532541.659664] EXT4-fs (sda4): unable to read superblock
[532541.660423] squashfs: SQUASHFS error: Can't find a SQUASHFS
superblock on sda4
[532541.661194] FAT-fs (sda4): bogus number of reserved sectors
[532541.661226] FAT-fs (sda4): Can't find a valid FAT filesystem
[532541.661901] isofs_fill_super: bread failed, dev=sda4,
iso_blknum=16, block=32
[532541.662588] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=256, location=256
[532541.662819] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532541.662879] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=512, location=512
[532541.662935] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532541.662994] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532541.663052] UDF-fs: warning (device sda4): udf_load_vrs: No anchor found
[532541.663086] UDF-fs: Rescanning with blocksize 2048
[532541.663124] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=256, location=256
[532541.663181] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=512, location=512
[532541.663238] UDF-fs: warning (device sda4): udf_load_vrs: No anchor found
[532541.663271] UDF-fs: warning (device sda4): udf_fill_super: No
partition found (1)
[532541.664182] XFS (sda4): Invalid superblock magic number
[532541.665014] (mount,17550,6):ocfs2_get_sector:1822 ERROR: status = -12
[532541.665048] (mount,17550,6):ocfs2_sb_probe:821 ERROR: status = -12
[532541.665081] (mount,17550,6):ocfs2_fill_super:1026 ERROR:
superblock probe failed!
[532541.665117] (mount,17550,6):ocfs2_fill_super:1217 ERROR: status = -12
[532541.665775] attempt to access beyond end of device
[532541.665806] sda4: rw=48, want=136, limit=2
[532541.665833] gfs2: error -5 reading superblock
[532541.666457] gfs2: gfs2 mount does not exist
[532541.667821] FAT-fs (sda4): bogus number of reserved sectors
[532541.667853] FAT-fs (sda4): Can't find a valid FAT filesystem
[532541.668621] ntfs: (device sda4): read_ntfs_boot_sector(): Primary
boot sector is invalid.
[532541.672594] ntfs: (device sda4): read_ntfs_boot_sector(): Mount
option errors=recover not used. Aborting without trying to recover.
[532541.672657] ntfs: (device sda4): ntfs_fill_super(): Not an NTFS volume.
[532603.588489] REISERFS warning (device sda4): sh-2006
read_super_block: bread failed (dev sda4, block 8, size 1024)
[532603.588552] REISERFS warning (device sda4): sh-2006
read_super_block: bread failed (dev sda4, block 64, size 1024)
[532603.588613] REISERFS warning (device sda4): sh-2021
reiserfs_fill_super: can not find reiserfs on sda4
[532603.589374] EXT3-fs (sda4): error: unable to read superblock
[532603.590054] EXT2-fs (sda4): error: unable to read superblock
[532603.590738] EXT4-fs (sda4): unable to read superblock
[532603.591524] squashfs: SQUASHFS error: Can't find a SQUASHFS
superblock on sda4
[532603.592590] FAT-fs (sda4): bogus number of reserved sectors
[532603.592622] FAT-fs (sda4): Can't find a valid FAT filesystem
[532603.593273] isofs_fill_super: bread failed, dev=sda4,
iso_blknum=16, block=32
[532603.593945] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=256, location=256
[532603.594247] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532603.594312] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=512, location=512
[532603.594371] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532603.594432] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532603.594492] UDF-fs: warning (device sda4): udf_load_vrs: No anchor found
[532603.594527] UDF-fs: Rescanning with blocksize 2048
[532603.594582] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=256, location=256
[532603.594642] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=512, location=512
[532603.594700] UDF-fs: warning (device sda4): udf_load_vrs: No anchor found
[532603.594746] UDF-fs: warning (device sda4): udf_fill_super: No
partition found (1)
[532603.595665] XFS (sda4): Invalid superblock magic number
[532603.596525] (mount,18660,7):ocfs2_get_sector:1822 ERROR: status = -12
[532603.596559] (mount,18660,7):ocfs2_sb_probe:821 ERROR: status = -12
[532603.596592] (mount,18660,7):ocfs2_fill_super:1026 ERROR:
superblock probe failed!
[532603.596627] (mount,18660,7):ocfs2_fill_super:1217 ERROR: status = -12
[532603.597271] attempt to access beyond end of device
[532603.597302] sda4: rw=48, want=136, limit=2
[532603.597329] gfs2: error -5 reading superblock
[532603.597944] gfs2: gfs2 mount does not exist
[532603.599274] FAT-fs (sda4): bogus number of reserved sectors
[532603.599306] FAT-fs (sda4): Can't find a valid FAT filesystem
[532603.600068] ntfs: (device sda4): read_ntfs_boot_sector(): Primary
boot sector is invalid.
[532603.600106] ntfs: (device sda4): read_ntfs_boot_sector(): Mount
option errors=recover not used. Aborting without trying to recover.
[532603.600169] ntfs: (device sda4): ntfs_fill_super(): Not an NTFS volume.
[532611.781265] REISERFS warning (device sda4): sh-2006
read_super_block: bread failed (dev sda4, block 8, size 1024)
[532611.781329] REISERFS warning (device sda4): sh-2006
read_super_block: bread failed (dev sda4, block 64, size 1024)
[532611.781389] REISERFS warning (device sda4): sh-2021
reiserfs_fill_super: can not find reiserfs on sda4
[532611.782171] EXT3-fs (sda4): error: unable to read superblock
[532611.782989] EXT2-fs (sda4): error: unable to read superblock
[532611.783768] EXT4-fs (sda4): unable to read superblock
[532611.784548] squashfs: SQUASHFS error: Can't find a SQUASHFS
superblock on sda4
[532611.785311] FAT-fs (sda4): bogus number of reserved sectors
[532611.785343] FAT-fs (sda4): Can't find a valid FAT filesystem
[532611.786003] isofs_fill_super: bread failed, dev=sda4,
iso_blknum=16, block=32
[532611.786658] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=256, location=256
[532611.786888] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532611.786957] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=512, location=512
[532611.787014] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532611.787072] UDF-fs: error (device sda4): udf_read_tagged: tag
version 0x0000 != 0x0002 || 0x0003, block 0
[532611.787131] UDF-fs: warning (device sda4): udf_load_vrs: No anchor found
[532611.787164] UDF-fs: Rescanning with blocksize 2048
[532611.787202] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=256, location=256
[532611.787259] UDF-fs: error (device sda4): udf_read_tagged: read
failed, block=512, location=512
[532611.787316] UDF-fs: warning (device sda4): udf_load_vrs: No anchor found
[532611.787349] UDF-fs: warning (device sda4): udf_fill_super: No
partition found (1)
[532611.788237] XFS (sda4): Invalid superblock magic number
[532611.789060] (mount,19654,5):ocfs2_get_sector:1822 ERROR: status = -12
[532611.789094] (mount,19654,5):ocfs2_sb_probe:821 ERROR: status = -12
[532611.789127] (mount,19654,5):ocfs2_fill_super:1026 ERROR:
superblock probe failed!
[532611.789162] (mount,19654,5):ocfs2_fill_super:1217 ERROR: status = -12
[532611.789800] attempt to access beyond end of device
[532611.789831] sda4: rw=48, want=136, limit=2
[532611.789858] gfs2: error -5 reading superblock
[532611.790487] gfs2: gfs2 mount does not exist
[532611.791860] FAT-fs (sda4): bogus number of reserved sectors
[532611.791892] FAT-fs (sda4): Can't find a valid FAT filesystem
[532611.792679] ntfs: (device sda4): read_ntfs_boot_sector(): Primary
boot sector is invalid.
[532611.792716] ntfs: (device sda4): read_ntfs_boot_sector(): Mount
option errors=recover not used. Aborting without trying to recover.
[532611.792780] ntfs: (device sda4): ntfs_fill_super(): Not an NTFS volume.

Thanks,
Kyle.


On Mon, Nov 16, 2015 at 5:53 AM, Kyle Sanderson <kyle.leet@xxxxxxxxx> wrote:
> I'll reboot the box tonight.
>
> I've been unable to run newer kernels (> 4.0) (gentoo/hardened-sources) on a
> variety different systems (amd64: atom, ivy bridge, haswell), they usually
> panic after a few hours. Primary issue being they're remote in a DC so I
> can't hook up a serial console.
>
> Thanks guys,
> Kyle.
>
> On 16 Nov 2015 12:54 a.m., "Thomas Gleixner" <tglx@xxxxxxxxxxxxx> wrote:
>>
>> On Sun, 15 Nov 2015, Linus Torvalds wrote:
>> > On Sun, Nov 15, 2015 at 2:28 PM, Kyle Sanderson <kyle.leet@xxxxxxxxx>
>> > wrote:
>> > > [] BUG: unable to handle kernel paging request at ffffe8ff7fc00001
>> > > [] IP: [<ffffffff810a174f>] kstat_irqs+0x4f/0x90
>> > > [] CPU: 2 PID: 1078 Comm: usage.pl Not tainted 4.1.7-hardened-r1 #1
>> > > [] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 1.0b
>> > > 04/21/2015
>> > RSI: 000060f700000001
>> > > [] Call Trace:
>> > > []  [<>] kstat_irqs_usr+0x1e/0x40
>>
>> > The code ends up being
>> >
>> >   mov    0x48(%r13),%rsi
>> >   mov    __per_cpu_offset(,%rcx,8),%rcx
>> >   add    (%rsi,%rcx,1),%ebx <-- trapping instruction
>> >
>> > which is just the
>> >
>> >    sum += *per_cpu_ptr(desc->kstat_irqs, cpu);
>> >
>> > part of kstat_irqs().
>> >
>> > Your registers being
>> >
>> >    RSI: 000060f700000001
>> >    RCX: ffff88087fc00000
>> >
>> > and it's RSI that makes no sense - RCX looks like a real kernel
>> > pointer. So it looks like it's the "desc->kstat_irqs" thing that is
>> > for some reason garbage.
>> >
>> > I don't see any sane possible reason this would happen, though.
>> > Thomas, does this look like anything you've seen before?
>>
>> No. What's strange is that this does explode while reading
>> /proc/interrupts and it did not happen when interrupt accounting took
>> place.
>>
>> Though this looks like memory corruption and it might be an interrupt
>> which fired only at boot time, i.e. before the corruption happened.
>>
>> No idea how to decode that. Kyle, is that reproducible?
>>
>> Thanks,
>>
>>         tglx
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/