Re: FS corruption after I/O errors

From: Vladislav Bolkhovitin
Date: Sat Nov 01 2008 - 09:08:53 EST

Next message: Mark Lord: "Re: PROBLEM: ioctl (HDIO_GETGEO) on partition"
Previous message: Peter Zijlstra: "[PATCH] ftrace: ftrace_dump_on_oops=[tracer]"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Vladislav Bolkhovitin wrote:

Nick Piggin wrote:
On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
Nick Piggin wrote:
On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
Hi,

During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed many

WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()

messages in kernel log on the initiator. I attached the full log of
several of them.

My target was buggy and I was working on fixing it, but I suppose Linux
should handle such failures more gracefully. In all the cases the target
had one type of failure: it "ate" a SCSI command and never returned
result of it.

Right. This is one of the warnings I see in my fault-injection testing.
It is fixed by my patch to clean up and improve the page and buffer
error handling in the vm/fs.

Can you specify which patch you referring? Is it in 2.6.27?
It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.

I see. I'm looking forward to see it in 2.6.28 or .29. This is really a needed work.

BTW, have you even seen in your fault-injection testing that after receiving a failure from a SCSI device during heavy load ext3 file system mounted on it gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps? I've many times seen that, including cases when the target was remaining up and fully functional. See, e.g., "MOANING MODE ON" part in http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked that case since then, although I see such corruptions quite often. But in all them I can't so clearly say that it isn't a target's failure.

I've just checked it with 2.6.27. The situation greatly improved and dbench was able to complete several runs under constant TASK_ABORTED "bombarding" (TASK RESET task management commands using "sg_reset -b" each 31 seconds from another "connection" to that device via qla2xxx initiator driver. You can see those resets in the attached log). But when then I unmounted the affected partition, e2fsck found errors on it. See attachments for details. The target all the times was fine and completely healthy.

[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdc: 69413/640000 files (0.2% non-contiguous), 316508/1280000 blocks
[root@inif w]# mount -t ext3 /dev/sdc /mnt
[root@inif dbench-mod]# while true; do ./dbench 50; done
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...................................+..........................................................................+.................................................................+.+.................................................+.....................................+................................................................................................................++...........................++....................++..........................+.......................+......................................+.................................................................................................+.........................++++.+...+................................+...++.+........++..........+.................++++......................+++....+.++.....+.....+.....................+......+++.++++**************************************************
Throughput 27.5109 MB/sec (NB=34.3887 MB/sec 275.109 MBit/sec)
Average latency 7.166755 ms
Maximum latency 22972.843000 ms
Latency > 2 sec. 0.091621 %
Latency > 10 sec. 0.016009 %
50 clients started
............................................................................................................write failed on handle 4290
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................................................................................................................................................................................................................................................................................................................................................................................++.........+......................................write failed on handle 4279
........................+.................................................+...................................................+...............+.........+...............................+.............++..............................++...........................+++........................+...+............+......+...+.+......+.....+...............++.....+....................+.....+..+++.+............+....+++......+.+...++.....+........+........+.+++++**************************************************
Throughput 27.5921 MB/sec (NB=34.4902 MB/sec 275.921 MBit/sec)
Average latency 7.097532 ms
Maximum latency 20892.153000 ms
Latency > 2 sec. 0.087833 %
Latency > 10 sec. 0.016437 %
50 clients started
....................................................................................write failed on handle 4290
..(4727) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4729) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4328 (File exists)
(4730) nb_close: handle 4328 was not open
(4739) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4743) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4745) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4329 (File exists)
(4751) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4330 (File exists)
(4752) nb_read: handle 4330 was not open size=24 ofs=2156
(4754) nb_read: handle 4330 was not open size=276 ofs=2139
(4755) nb_read: handle 4330 was not open size=14 ofs=2136
(4757) nb_read: handle 4330 was not open size=7 ofs=2136
(4759) nb_read: handle 4330 was not open size=12 ofs=2138
(4761) nb_read: handle 4330 was not open size=8 ofs=2137
(4763) nb_read: handle 4330 was not open size=15 ofs=2135
(4765) nb_read: handle 4330 was not open size=16 ofs=0
(4767) nb_read: handle 4330 was not open size=2048 ofs=16
(4768) nb_read: handle 4330 was not open size=2048 ofs=2064
(4769) nb_read: handle 4330 was not open size=2048 ofs=257
(4770) nb_read: handle 4330 was not open size=2048 ofs=385
(4771) nb_read: handle 4330 was not open size=2048 ofs=513
(4772) nb_read: handle 4330 was not open size=2048 ofs=641
(4773) nb_read: handle 4330 was not open size=2048 ofs=769
(4774) nb_read: handle 4330 was not open size=2048 ofs=897
(4775) nb_read: handle 4330 was not open size=2048 ofs=1025
(4776) nb_read: handle 4330 was not open size=2048 ofs=1153
(4777) nb_read: handle 4330 was not open size=2048 ofs=1281
(4778) nb_read: handle 4330 was not open size=2048 ofs=1409
(4779) nb_read: handle 4330 was not open size=408 ofs=1537
..(4899) nb_close: handle 4330 was not open
(4964) nb_read: handle 4329 was not open size=1386 ofs=2157
(4971) nb_read: handle 4329 was not open size=4096 ofs=2157
(4972) nb_read: handle 4329 was not open size=4096 ofs=6253
(4973) nb_read: handle 4329 was not open size=4096 ofs=10349
(4974) nb_read: handle 4329 was not open size=4096 ofs=14445
(4975) nb_read: handle 4329 was not open size=4096 ofs=18541
(4976) nb_read: handle 4329 was not open size=256 ofs=22637
(4977) nb_read: handle 4329 was not open size=4096 ofs=22893
(4978) nb_read: handle 4329 was not open size=4096 ofs=26989
(4979) nb_read: handle 4329 was not open size=4096 ofs=31085
(4980) nb_read: handle 4329 was not open size=4096 ofs=35181
(4981) nb_read: handle 4329 was not open size=4096 ofs=39277
(4982) nb_read: handle 4329 was not open size=4096 ofs=43373
(4983) nb_read: handle 4329 was not open size=4096 ofs=47469
(4986) nb_read: handle 4329 was not open size=1024 ofs=2157
(4987) nb_read: handle 4329 was not open size=4688 ofs=1667
(4988) nb_read: handle 4329 was not open size=3516 ofs=6355
(4989) nb_read: handle 4329 was not open size=16384 ofs=9871
(4990) nb_read: handle 4329 was not open size=3771 ofs=26255
(4991) nb_read: handle 4329 was not open size=2048 ofs=30026
(4992) nb_read: handle 4329 was not open size=1024 ofs=32074
(4993) nb_read: handle 4329 was not open size=6652 ofs=33098
(4994) nb_read: handle 4329 was not open size=16384 ofs=39750
(4995) nb_read: handle 4329 was not open size=10496 ofs=56134
(4996) nb_read: handle 4329 was not open size=1784 ofs=66630
(4997) nb_read: handle 4329 was not open size=90 ofs=68414
(4998) nb_read: handle 4329 was not open size=273 ofs=68504
(4999) nb_read: handle 4329 was not open size=1600 ofs=68777
(5000) nb_read: handle 4329 was not open size=640 ofs=70377
(5001) nb_read: handle 4329 was not open size=1280 ofs=71017
(5002) nb_read: handle 4329 was not open size=8464 ofs=72297
(5003) nb_read: handle 4329 was not open size=158 ofs=80761
(5004) nb_read: handle 4329 was not open size=912 ofs=80919
(5005) nb_read: handle 4329 was not open size=1980 ofs=81831
(5006) nb_read: handle 4329 was not open size=2256 ofs=83811
(5007) nb_read: handle 4329 was not open size=400 ofs=86067
(5008) nb_read: handle 4329 was not open size=252 ofs=86467
(5009) nb_read: handle 4329 was not open size=42 ofs=86719
(5011) nb_read: handle 4329 was not open size=725 ofs=86761
(5012) nb_read: handle 4329 was not open size=512 ofs=2009
(5013) nb_read: handle 4329 was not open size=9 ofs=3895
(5015) nb_read: handle 4329 was not open size=46 ofs=3781
(5017) nb_read: handle 4329 was not open size=92 ofs=4059
(5018) nb_read: handle 4329 was not open size=92 ofs=3781
(5019) nb_read: handle 4329 was not open size=184 ofs=3986
(5020) nb_read: handle 4329 was not open size=16384 ofs=3987
(5021) nb_read: handle 4329 was not open size=16384 ofs=3902
(5022) nb_read: handle 4329 was not open size=10240 ofs=3781
(5023) nb_read: handle 4329 was not open size=16384 ofs=3821
(5024) nb_read: handle 4329 was not open size=935 ofs=3891
(5025) nb_read: handle 4329 was not open size=10240 ofs=4060
(5026) nb_read: handle 4329 was not open size=184 ofs=4059
(5027) nb_read: handle 4329 was not open size=184 ofs=256
(5028) nb_read: handle 4329 was not open size=1026 ofs=256
(5029) nb_read: handle 4329 was not open size=326 ofs=3901
(5030) nb_read: handle 4329 was not open size=652 ofs=3899
(5031) nb_read: handle 4329 was not open size=2 ofs=256
(5033) nb_read: handle 4329 was not open size=4948 ofs=258
(5034) nb_close: handle 4329 was not open
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...+......................................................................+..................................+.............................+.....................+++.........................................................................................................................+....................+...............+.....................................+......+......+.+...............................+.+....+.......................+.................+..................++....+.+..+..........++..+...........+.........+.........+..........................++++........+++++.............................+.++...........+..++++++**************************************************
Throughput 27.8199 MB/sec (NB=34.7749 MB/sec 278.199 MBit/sec)
Average latency 6.940010 ms
Maximum latency 18993.291000 ms
Latency > 2 sec. 0.081048 %
Latency > 10 sec. 0.015581 %
50 clients started
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+.................................................................+....................................................................................................................................................+..................+...+.............................................................................................................................................................................................................+..................+.......................................................+................++.......+............+.....+...............................+..+...............++.+.................................+............+.....++..++.......................+.........+...............++...........................+++.........+++.+...............+....+...+.....+........+......++....++++++.++**************************************************
Throughput 27.0223 MB/sec (NB=33.7779 MB/sec 270.223 MBit/sec)
Average latency 6.739319 ms
Maximum latency 18699.530000 ms
Latency > 2 sec. 0.090834 %
Latency > 10 sec. 0.013650 %
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................................+............................................................................+...................................................................................................................................................................+.+................................................................+.................................................++++...........................+..+....+..........++...........+....................+.....+.............................................+.............+................+...+..........+++....+....+............................+........+..+.....+....+...+........+........+++...............+++++..++++.++++**************************************************
Throughput 27.2861 MB/sec (NB=34.1077 MB/sec 272.861 MBit/sec)
Average latency 7.051013 ms
Maximum latency 23801.490000 ms
Latency > 2 sec. 0.091263 %
Latency > 10 sec. 0.014722 %
50 clients started
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
[root@inif w]# umount /mnt
[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(1250434--1250435) -(1273360--1273370)
Fix? yes

Free blocks count wrong for group #38 (32253, counted=32266).
Fix? yes

Free blocks count wrong (1003250, counted=1003263).
Fix? yes

/dev/sdc: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdc: 68842/640000 files (0.1% non-contiguous), 276737/1280000 blocks
Nov 1 16:27:35 inif kernel: kjournald starting. Commit interval 5 seconds
Nov 1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal
Nov 1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352
Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808
Nov 1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0
Nov 1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432
Nov 1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792
Nov 1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720
Nov 1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752
Nov 1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288
Nov 1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536
Nov 1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784
Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776
Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0
Nov 1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544
Nov 1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480
Nov 1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840
Nov 1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0
Nov 1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128
Nov 1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064
Nov 1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504
Nov 1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400
Nov 1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376
Nov 1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192
Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872
Nov 1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760
Nov 1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864
Nov 1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded

Next message: Mark Lord: "Re: PROBLEM: ioctl (HDIO_GETGEO) on partition"
Previous message: Peter Zijlstra: "[PATCH] ftrace: ftrace_dump_on_oops=[tracer]"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]