Re: Strange block/scsi/workqueue issue

From: James Bottomley
Date: Tue Apr 12 2011 - 12:27:14 EST


On Tue, 2011-04-12 at 17:04 +0100, Steven Whitehouse wrote:
> Hi,
>
> On Tue, 2011-04-12 at 10:14 -0500, James Bottomley wrote:
> > On Tue, 2011-04-12 at 15:06 +0100, Steven Whitehouse wrote:
> > > Messages attached - getting closer I think, but still not quite there.
> > > It is very odd that so far (at least I've seen no other reports) that I
> > > seem to be the only one who hits this. I wonder what is different about
> > > my kernel/hardware...
> >
> >
> > > csi: killing requests for dead queue
> > > scsi: killing requests for dead queue
> > > BUG: unable to handle kernel scsi: killing requests for dead queue
> > > NULL pointer dereference at 0000000000000028
> > > IP: [<ffffffff813b34ed>] blk_peek_request+0x1ad/0x220
> >
> > Hmm, I don't see where this is ... it's a null pointer deref offset by
> > 0x28 bytes ... but I can't see where ... can you resolve this to a line?
> >
> > James
> >
> >
>
> So assuming I've calculated this correctly, it should be just about....
>
> /home/steve/linux-2.6/include/linux/rcupdate.h:677
> 6de8: a8 08 test $0x8,%al
> 6dea: 0f 84 65 ff ff ff je 6d55 <blk_peek_request+0xf5>
> 6df0: e8 00 00 00 00 callq 6df5 <blk_peek_request+0x195>
> 6df1: R_X86_64_PC32 preempt_schedule-0x4
> 6df5: e9 5b ff ff ff jmpq 6d55 <blk_peek_request+0xf5>
> 6dfa: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
> __elv_next_request():
> /home/steve/linux-2.6/block/blk.h:65
> if (!list_empty(&q->queue_head)) {
> rq = list_entry_rq(q->queue_head.next);
> return rq;
> }
>
> if (!q->elevator->ops->elevator_dispatch_fn(q, 0))
> 6e00: 49 8b 44 24 18 mov 0x18(%r12),%rax
> 6e05: 31 f6 xor %esi,%esi
> 6e07: 48 8b 00 mov (%rax),%rax
> 6e0a: 4c 89 e7 mov %r12,%rdi <---- here!
> 6e0d: ff 50 28 callq *0x28(%rax)
> 6e10: 85 c0 test %eax,%eax
> 6e12: 0f 85 f4 fe ff ff jne 6d0c <blk_peek_request+0xac>
> 6e18: 45 31 ed xor %r13d,%r13d
> 6e1b: e9 70 ff ff ff jmpq 6d90 <blk_peek_request+0x130>
> blk_peek_request():
> /home/steve/linux-2.6/block/blk-core.c:1912
> * so that we don't add it again
> */
>
> The offset I got was 6e0d, but I guess thats the start of the following instruction
> by the time the fault is logged. Also I've attached my current .config just in case
> that might be a help is reproducing it,

Good work! it's simpler if you compile with -g ... then the gdb line
command will just tell you all of this. However, it seems clear that
elevator->ops is NULL (elevator_dispatch_fn is 0x28 offset into that in
a 64 bit system).

That's because blk_cleanup_queue called elevator_exit, so we need an out
in the peek case.

So unwrapping the onion, this is the next level

James

---

Index: linux-2.6/block/blk.h
===================================================================
--- linux-2.6.orig/block/blk.h
+++ linux-2.6/block/blk.h
@@ -62,7 +62,7 @@ static inline struct request *__elv_next
return rq;
}

- if (!q->elevator->ops->elevator_dispatch_fn(q, 0))
+ if (!q->elevator->ops || !q->elevator->ops->elevator_dispatch_fn(q, 0))
return NULL;
}
}


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/