Re: 2.4.30-hf1 do_IRQ stack overflows
From: Manfred Schwarb
Date: Fri Jun 10 2005 - 03:12:23 EST
> Hi,
>
> On Tue, Jun 07, 2005 at 02:38:01PM +0200, Manfred Schwarb wrote:
> >
> >
> > >
> > > Hi Manfred,
> > >
> > > On Wed, May 11, 2005 at 10:15:02AM +0200, Manfred Schwarb wrote:
> > > > Hi,
> > > > with recent versions of the 2.4 kernel (Vanilla), I get an
> increasing
> > > amount of do_IRQ stack overflows.
> > > > This night, I got 3 of them.
> > > > With 2.4.28 I got an overflow about twice a year, with 2.4.29 nearly
> > > once a month and with
> > > > 2.4.30 nearly every day 8-((
> > >
> > > The system is getting dangerously close to an actual stack overflow,
> which
> > > would
> > > crash the system.
> > >
> > > "do_IRQ: stack overflow: " indicates how many bytes are still
> available.
> > >
> > > The traces show huge networking execution paths.
> > >
> > > It seems you are using some packet scheduler (CONFIG_NET_SCHED)?
> Pretty
> > > much all
> > > traces show functions from sch_generic.c. Can you disable that for a
> test?
> > >
> >
> > Sorry to bother you again, but the problem didn't vanish completely.
> > This morning, I caught another one. I built a new kernel with
> > CONFIG_NET_SCHED=n as suggested, uptime is now 25 days, and the
> following
> > is the first do_IRQ since then (ksymoops -i):
> >
> > Jun 7 03:55:01 tp-meteodat7 kernel: f3238830 00000280 f49e7b80 00000000
<---------snip-------->
> > [<c01594d4>] [<c01598d9>] [<c0159bb4>] [<c0158905>]
> > Warning (Oops_read): Code line not seen, dumping what data is available
>
> Do you have the "do_IRQ stack overflow" output and the amount of bytes
> left it informs?
>
Yes, it was a close one, 640. I append the original output to the end of
this email. Thanks for looking at this.
> > Trace; c010d948 <call_do_IRQ+5/d>
> > Trace; c023a039 <skb_copy_and_csum_dev+49/100>
<---------snip-------->
> > Trace; c010d948 <call_do_IRQ+5/d>
>
> I dont see any huge stack consumers on this callchain.
>
> David, Herbert, any clues what might be going on here?
>
>
Jun 7 03:55:01 tp-meteodat7 kernel: do_IRQ: stack overflow: 640
Jun 7 03:55:01 tp-meteodat7 kernel: f3238830 00000280 f49e7b80 00000000
00000042 cca1388e f4116980 f17aa000
Jun 7 03:55:01 tp-meteodat7 kernel: c010d948 00000042 f4116980
00000000 cca1388e f4116980 f17aa000 00000042
Jun 7 03:55:01 tp-meteodat7 kernel: 00000018 f61d0018 ffffff14
c023a039 00000010 00000246 ee5ea480 00000000
Jun 7 03:55:01 tp-meteodat7 kernel: Call Trace: [call_do_IRQ+5/13]
[skb_copy_and_csum_dev+73/256]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256445916/96]
[qdisc_restart+114/432] [dev_queue_xmit+383/880]
Jun 7 03:55:01 tp-meteodat7 kernel: Call Trace: [<c010d948>]
[<c023a039>] [<f90df5dc>] [<c0248402>] [<c023cc7f>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_finish_output2+184/336]
[ip_finish_output2+0/336] [ip_finish_output2+0/336] [nf_hook_slow+478/528]
[ip_finish_output2+0/336] [ip_output+334/480]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c02561a8>] [<c02560f0>]
[<c02560f0>] [<c024760e>] [<c02560f0>] [<c025492e>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_finish_output2+0/336]
[ip_queue_xmit2+213/671] [ip_queue_xmit2+0/671] [ip_queue_xmit2+0/671]
[nf_hook_slow+478/528] [ip_queue_xmit2+0/671]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c02560f0>] [<c0256315>]
[<c0256240>] [<c0256240>] [<c024760e>] [<c0256240>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_queue_xmit+845/1536]
[ip_queue_xmit2+0/671] [tcp_v4_send_check+160/240]
[tcp_transmit_skb+1001/1792] [tcp_send_ack+132/208] [tcp_rfree+0/32]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c0254d0d>] [<c0256240>]
[<c026daf0>] [<c0267c99>] [<c026a6f4>] [<c0259370>]
Jun 7 03:55:01 tp-meteodat7 kernel: [tcp_rfree+0/32]
[tcp_rcv_established+2042/2640] [tcp_v4_do_rcv+314/352]
[tcp_v4_rcv+1726/1952] [ip_local_deliver_finish+351/416]
[ip_local_deliver_finish+0/416]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c0259370>] [<c02661ca>]
[<c026edaa>] [<c026f48e>] [<c025174f>] [<c02515f0>]
Jun 7 03:55:01 tp-meteodat7 kernel: [nf_hook_slow+478/528]
[ip_local_deliver_finish+0/416] [ip_rcv_finish+0/616]
[ip_local_deliver+399/576] [ip_local_deliver_finish+0/416]
[ip_rcv_finish+0/616]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c024760e>] [<c02515f0>]
[<c0251790>] [<c02510df>] [<c02515f0>] [<c0251790>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_rcv_finish+473/616]
[ip_rcv_finish+0/616] [nf_hook_slow+478/528] [ip_rcv_finish+0/616]
[ip_rcv+808/1120] [ip_rcv_finish+0/616]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c0251969>] [<c0251790>]
[<c024760e>] [<c0251790>] [<c02514b8>] [<c0251790>]
Jun 7 03:55:01 tp-meteodat7 kernel: [netif_receive_skb+485/544]
[process_backlog+147/304] [net_rx_action+250/368] [do_softirq+118/224]
[do_IRQ+244/304] [call_do_IRQ+5/13]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c023d4d5>] [<c023d5a3>]
[<c023d73a>] [<c01254c6>] [<c010b094>] [<c010d948>]
Jun 7 03:55:01 tp-meteodat7 kernel: [set_ldt_desc+5/59]
[schedule+650/1344] [schedule_timeout+84/160] [process_timeout+0/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256523927/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256524281/96]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c010a625>] [<c011ce8a>]
[<c011cb14>] [<c011ca60>] [<f90f2697>] [<f90f27f9>]
Jun 7 03:55:01 tp-meteodat7 kernel:
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256524459/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256530630/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256904584/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256908606/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256911957/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250526660/96]
Jun 7 03:55:01 tp-meteodat7 kernel: [<f90f28ab>] [<f90f40c6>]
[<f914f588>] [<f915053e>] [<f9151255>] [<f8b3a3c4>]
Jun 7 03:55:01 tp-meteodat7 kernel:
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250598640/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250526660/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250605583/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250526660/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250526660/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250602612/96]
Jun 7 03:55:01 tp-meteodat7 kernel: [<f8b4bcf0>] [<f8b3a3c4>]
[<f8b4d80f>] [<f8b3a3c4>] [<f8b3a3c4>] [<f8b4cc74>]
Jun 7 03:55:01 tp-meteodat7 kernel:
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250526660/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250525224/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250568899/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250613955/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250576507/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250526472/96]
Jun 7 03:55:01 tp-meteodat7 kernel: [<f8b3a3c4>] [<f8b39e28>]
[<f8b448c3>] [<f8b4f8c3>] [<f8b4667b>] [<f8b3a308>]
Jun 7 03:55:01 tp-meteodat7 kernel:
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250576301/96]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4250628604/96]
[__kfree_skb+246/336]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256445940/96]
[qdisc_restart+31/432] [dev_queue_xmit+383/880]
Jun 7 03:55:01 tp-meteodat7 kernel: [<f8b465ad>] [<f8b531fc>]
[<c02387b6>] [<f90df5f4>] [<c02483af>] [<c023cc7f>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_finish_output2+184/336]
[__kfree_skb+246/336]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256445940/96]
[__kfree_skb+246/336]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256445940/96]
[qdisc_restart+31/432]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c02561a8>] [<c02387b6>]
[<f90df5f4>] [<c02387b6>] [<f90df5f4>] [<c02483af>]
Jun 7 03:55:01 tp-meteodat7 kernel: [dev_queue_xmit+383/880]
[__kfree_skb+246/336]
[nfsd:__insmod_nfsd_O/lib/modules/2.4.30-hf1/kernel/fs/nfsd/nfsd.+4256445940/96]
[qdisc_restart+31/432] [dev_queue_xmit+383/880] [ip_finish_output2+184/336]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_finish_output2+0/336]
[ip_finish_output2+0/336] [nf_hook_slow+478/528] [ip_finish_output2+0/336]
[ip_output+334/480] [ip_finish_output2+0/336]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c02560f0>] [<c02560f0>]
[<c024760e>] [<c02560f0>] [<c025492e>] [<c02560f0>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_queue_xmit2+213/671]
[sock_def_readable+99/128] [tcp_rfree+0/32] [tcp_rfree+0/32]
[tcp_rcv_established+1981/2640] [tcp_v4_do_rcv+314/352]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c0256315>] [<c0237f63>]
[<c0259370>] [<c0259370>] [<c026618d>] [<c026edaa>]
Jun 7 03:55:01 tp-meteodat7 kernel: [tcp_v4_rcv+1726/1952]
[ip_local_deliver_finish+351/416] [ip_local_deliver_finish+0/416]
[nf_hook_slow+478/528] [ip_local_deliver_finish+0/416] [ip_rcv_finish+0/616]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c026f48e>] [<c025174f>]
[<c02515f0>] [<c024760e>] [<c02515f0>] [<c0251790>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_local_deliver+399/576]
[ip_local_deliver_finish+0/416] [ip_rcv_finish+0/616]
[ip_rcv_finish+473/616] [ip_rcv_finish+0/616] [nf_hook_slow+478/528]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c02510df>] [<c02515f0>]
[<c0251790>] [<c0251969>] [<c0251790>] [<c024760e>]
Jun 7 03:55:01 tp-meteodat7 kernel: [ip_rcv_finish+0/616]
[__alloc_pages+100/640] [poll_freewait+68/80] [do_select+569/592]
[sys_select+660/1248] [sys_ioctl+677/785]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c0251790>] [<c013e624>]
[<c01594d4>] [<c01598d9>] [<c0159bb4>] [<c0158905>]
Jun 7 03:55:01 tp-meteodat7 kernel: [sys_time+21/80] [system_call+51/56]
Jun 7 03:55:01 tp-meteodat7 kernel: [<c0124bb5>] [<c0108fa7>]
In my previous posting, I snipped /var/log/messages at the wrong place,
and zapped the last line. This line results in two additional
lines of the ksymoops output:
Trace; c0124bb5 <sys_time+15/50>
Trace; c0108fa7 <system_call+33/38>
--
Geschenkt: 3 Monate GMX ProMail gratis + 3 Ausgaben stern gratis
++ Jetzt anmelden & testen ++ http://www.gmx.net/de/go/promail ++
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/