[SCHED] rcu_sched kthread starved for N jiffies! + stack trace

[Fabrice Triboix]

Dear All,

I did some soak testing of our application (running on linux) over the weekend. Our application runs on a dedicated linux system.
This morning, I saw in "/var/log/message" the following message repeated 70 times "rcu_sched kthread starved for N jiffies!" (N being a number in the tens of thousands). It also comes with a stack trace. Please find a copy/paste of the latest error message below.

For information, we are using SCHED_FIFO for some threads that needs to run with very low-latency (video card captures).

If anyone could shed some light on what that means, that would be greatly appreciated!

Many thanks,

  Fabrice Triboix


    ftenc:~# grep kernel /var/log/messages | tail -30
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369148]  [<ffffffff81887e0f>] apic_timer_interrupt+0x6f/0x80
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369149]  <EOI>  [<ffffffff81887b42>] ? retint_restore_args+0xe/0xe
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369154]  [<ffffffff8100cae5>] ? default_idle+0x25/0xb0
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369156]  [<ffffffff8100cae3>] ? default_idle+0x23/0xb0
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369158]  [<ffffffff8100d42f>] arch_cpu_idle+0xf/0x20
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369162]  [<ffffffff810f64a5>] cpu_startup_entry+0x2e5/0x400
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369164]  [<ffffffff81127103>] ? clockevents_register_device+0xe3/0x140
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369167]  [<ffffffff8112718c>] ? clockevents_config_and_register+0x2c/0x40
    Nov 30 09:13:33 plink-buildroot user.warn kernel: [332879.369169]  [<ffffffff8103448a>] start_secondary+0x13a/0x150
    Nov 30 09:13:33 plink-buildroot user.err kernel: [332879.369171] rcu_sched kthread starved for 20999 jiffies!
    Nov 30 09:32:14 plink-buildroot user.err kernel: [334000.974890] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 5, t=21002 jiffies, g=6612437, c=6612436, q=11267)
    Nov 30 09:32:14 plink-buildroot user.err kernel: [334000.974900] All QSes seen, last rcu_sched kthread activity 20999 (4628342351-4628321352), jiffies_till_next_fqs=3
    Nov 30 09:32:14 plink-buildroot user.info kernel: [334000.974903] mux-0           R  running task        0 10129   8797 0x00000000
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974907]  ffffffff81e4c3c0 ffff88089fc83dd8 ffffffff810e53ef ffff88089fc943c0
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974909]  ffffffff81e4c3c0 ffff88089fc83e48 ffffffff811157cc ffff880837edbf00
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974911]  ffff88089fc943c0 0000000000002c03 000000000064e5d5 0000000000000000
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974914] Call Trace:
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974916]  <IRQ>  [<ffffffff810e53ef>] sched_show_task+0xaf/0x120
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974934]  [<ffffffff811157cc>] rcu_check_callbacks+0x70c/0x780
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974937]  [<ffffffff8111a5e9>] update_process_times+0x39/0x60
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974942]  [<ffffffff811296f6>] tick_sched_handle.isra.16+0x36/0x50
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974945]  [<ffffffff81129754>] tick_sched_timer+0x44/0x80
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974947]  [<ffffffff8111b2f0>] __run_hrtimer+0x80/0x1c0
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974950]  [<ffffffff81129710>] ? tick_sched_handle.isra.16+0x50/0x50
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974952]  [<ffffffff8111b693>] hrtimer_interrupt+0xf3/0x210
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974958]  [<ffffffff81035c99>] local_apic_timer_interrupt+0x39/0x60
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974964]  [<ffffffff81889a25>] smp_apic_timer_interrupt+0x45/0x60
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974967]  [<ffffffff81887e0f>] apic_timer_interrupt+0x6f/0x80
    Nov 30 09:32:14 plink-buildroot user.warn kernel: [334000.974968]  <EOI>  [<ffffffff81887abe>] ? retint_swapgs+0xe/0x44
    Nov 30 09:32:14 plink-buildroot user.err kernel: [334000.974971] rcu_sched kthread starved for 20999 jiffies!
    ftenc:~#
    ftenc:~#
    ftenc:~# uname -a
    Linux plink-buildroot 4.0.0 #2 SMP Thu Nov 26 10:34:48 GMT 2015 x86_64 GNU/Linux



Company registration number: 08121803
Place of registration: England

This email and any attachments are sent in strictest confidence for the sole use of the addressee
and may contain legally privileged, confidential, and proprietary data. If you are not the intended
recipient, please advise the sender by replying promptly to this email and then delete and destroy
this email and any attachments without any further use, copying or forwarding.

Please consider your environmental responsibility before printing this e-mail
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/