Re: [OOPS] 2.6.21-rc6-git5 in cfq_dispatch_insert

[Roland Kuhn]

Hi Jens!

[I made a typo in the Cc: list so that lkml is only included as of  
now. Actually I copied the typo from you ;-) ]

On 24 Apr 2007, at 11:40, Jens Axboe wrote:

> On Tue, Apr 24 2007, Jens Axboe wrote:
> > On Tue, Apr 24 2007, Roland Kuhn wrote:
> > > Hi Jens!
> > >
> > > On 24 Apr 2007, at 11:18, Jens Axboe wrote:
> > >
> > > > On Tue, Apr 24 2007, Roland Kuhn wrote:
> > > > > Hi Jens!
> > > > >
> > > > > We're using a custom built fileserver (dual core Athlon64, using
> > > > > x64_64 arch) with 22 disks in a RAID6 and while resyncing /dev/md2
> > > > > (9.1GB ext3) after a hardware incident (cable pulled on one  
> > > > > disk) the
> > > > > machine would reliably oops while serving some large files over
> > > > > NFSv3. The oops message scrolled partly off the screen, but the IP
> > > > > was in cfq_dispatch_insert, so I tried your debug patch from
> > > > > yesterday with 2.6.21-rc7. I used netconsole for capturing the  
> > > > > output
> > > > > (which works nicely, thanks Matt!) and as usual the condition
> > > > > triggered after about half a minute, this with the following  
> > > > > printout
> > > > > instead of crashing (still works fine):
> > > > >
> > > > > cfq: rbroot not empty, but ->next_rq == NULL! Fixing up, report  
> > > > > the
> > > > > issue to lkml@xxxxxxxxxxxxxxx
> > > > > cfq: busy=1,drv=1,timer=0
> > > > > cfq rr_list:
> > > > > cfq busy_list:
> > > > >  4272: sort=0,next=0000000000000000,q=0/1,a=2/0,d=0/1,f=221
> > > > > cfq idle_list:
> > > > > cfq cur_rr:
> > > > > cfq: rbroot not empty, but ->next_rq == NULL! Fixing up, report  
> > > > > the
> > > > > issue to lkml@xxxxxxxxxxxxxxx
> > > > > cfq: busy=1,drv=1,timer=0
> > > > > cfq rr_list:
> > > > > cfq busy_list:
> > > > >  4276: sort=0,next=0000000000000000,q=0/1,a=2/0,d=0/1,f=221
> > > > > cfq idle_list:
> > > > > cfq cur_rr:
> > > > >
> > > > > There was no backtrace, so the only thing I can tell is that  
> > > > > for the
> > > > > previous crashes some nfs threads were always involved, only  
> > > > > once did
> > > > > it happen inside an interrupt handler (with the "aieee" kind of
> > > > > message).
> > > > >
> > > > > If you want me to try something else, don't hesitate to ask!
> > > >
> > > > Nifty, great that you can reproduce so quickly. I'll try a 3-drive
> > > > raid6
> > > > here and see if read activity along with a resync will trigger
> > > > anything.
> > > > If that doesn't work for me, I'll provide you with a more extensive
> > > > debug patch (if you don't mind).
> > > Sure. You might want to include NFS file access into your tests,
> > > since we've not triggered this with locally accessing the disks.  
> > > BTW:
> >
> > How are you exporting the directory (what exports options) - how  
> > is it
> > mounted by the client(s)? What chunksize is your raid6 using?
>
> And what are the nature of the files on the raid (huge, small, ?) and
> what are the client(s) doing? Just approximately, I know these things
> can be hard/difficult/impossible to specify.
The files are 100-400MB in size and the client is merging them into a  
new file in the same directory using the ROOT library, which does in  
essence alternating sequences of

_llseek(somewhere)
read(n bytes)
_llseek(somewhere+n)
read(m bytes)
...

and then

_llseek(somewhere)
rt_sigaction(ignore INT)
write(n bytes)
rt_sigaction(INT->DFL)
time()
_llseek(somewhere+n)
...

where n is of the the order of 30kB. The input files are treated  
sequentially, not randomly.

BTW: the machine just stopped dead, no sign whatsoever on console or  
netconsole, so I rebooted with elevator=deadline
(need to get some work done besides ;-) )

Ciao,
                    Roland

--
TU Muenchen, Physik-Department E18, James-Franck-Str., 85748 Garching
Telefon 089/289-12575; Telefax 089/289-12570
--
CERN office: 892-1-D23 phone: +41 22 7676540 mobile: +41 76 487 4482
--
Any society that would give up a little liberty to gain a little
security will deserve neither and lose both.  - Benjamin Franklin
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P+++ L+++ E(+) W+ !N K- w--- M 
+ !V Y+
PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- y+++
------END GEEK CODE BLOCK------

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Attachment: PGP.sig
Description: This is a digitally signed message part