v4.4-rc3: rcu_sched self-detected stall

[Andy Shevchenko]

Get 100% reproducible result on 4.4-rc3 on Intel BayTrail platform

Any suggestions?

P.S. Something like that was on 4.1-rc7 (same kernel config), though
didn't gather the traceback.

[ÂÂÂÂ0.000000] Command line: vmlinuz.efi initrd=initrd
console=ttyS0,115200n8
Â
[ÂÂÂÂ3.680557] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
0x13381e82524, max_idle_ns: 440795248224 ns
[ÂÂÂ23.938953] INFO: rcu_sched self-detected stall on CPU[ÂÂÂ23.939947]
INFO: rcu_sched detected stalls on CPUs/ta
sks:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
[ÂÂÂ23.939955]ÂÂ3-...: (20996 ticks this GP) idle=4e3/140000000000001/0
softirq=203/203 fqs=589 
[ÂÂÂ23.939956]ÂÂ(detected by 0, t=21002 jiffies, g=-173, c=-174, q=90)
[ÂÂÂ23.939962] Task dump for CPU 3:
[ÂÂÂ23.939964] swapper/0ÂÂÂÂÂÂÂRÂÂrunning taskÂÂÂÂ13280ÂÂÂÂÂ1ÂÂÂÂÂÂ0
0x00000008
[ÂÂÂ23.939975]ÂÂ0000000000000286 ffff880036e97bd8 ffffffff8130528d
0000000000000001
[ÂÂÂ23.939980]ÂÂffff880036c2c810 00000000025000c0 0000000000000202
0000000000000202
[ÂÂÂ23.939984]ÂÂffff880036e97c20 ffffffff813053a5 0000241b8115f5db
0000241b811d77ec
[ÂÂÂ23.939989] Call Trace:
[ÂÂÂ23.940002]ÂÂ[<ffffffff8130528d>] ? ida_pre_get+0x4d/0xe0
[ÂÂÂ23.940006]ÂÂ[<ffffffff813053a5>] ? ida_simple_get+0x85/0xe0
[ÂÂÂ23.940012]ÂÂ[<ffffffff811d8815>] ? kernfs_activate+0x75/0xe0
[ÂÂÂ23.940016]ÂÂ[<ffffffff811d897e>] ? kernfs_add_one+0xfe/0x160
[ÂÂÂ23.940021]ÂÂ[<ffffffff811d9f1a>] ? __kernfs_create_file+0x6a/0x90
[ÂÂÂ23.940025]ÂÂ[<ffffffff811da6c4>] ?
sysfs_add_file_mode_ns+0x94/0x180
[ÂÂÂ23.940029]ÂÂ[<ffffffff81306e45>] ? kobject_uevent_env+0x75/0x640
[ÂÂÂ23.940034]ÂÂ[<ffffffff8141cca6>] ? klist_children_get+0x16/0x20
[ÂÂÂ23.940039]ÂÂ[<ffffffff81013760>] ? subcaches_show+0x40/0x40
[ÂÂÂ23.940045]ÂÂ[<ffffffff810be900>]
?smp_call_function_single+0xb0/0x100
[ÂÂÂ23.940049]ÂÂ[<ffffffff810be8f6>] ?
smp_call_function_single+0xa6/0x100
[ÂÂÂ23.940053]ÂÂ[<ffffffff81013760>] ? subcaches_show+0x40/0x40
[ÂÂÂ23.940057]ÂÂ[<ffffffff810147ed>] ? init_cache_level+0x1d/0x30
[ÂÂÂ23.940062]ÂÂ[<ffffffff81427119>] ?
detect_cache_attributes+0x19/0x1a0
[ÂÂÂ23.940065]ÂÂ[<ffffffff814273f9>] ? cache_add_dev+0x159/0x1f0
[ÂÂÂ23.940071]ÂÂ[<ffffffff81f8cade>] ? container_dev_init+0x2d/0x2d
[ÂÂÂ23.940074]ÂÂ[<ffffffff81f8cb1e>] ? cacheinfo_sysfs_init+0x40/0x8a
[ÂÂÂ23.940078]ÂÂ[<ffffffff810003d6>] ? do_one_initcall+0xa6/0x1c0
[ÂÂÂ23.940083]ÂÂ[<ffffffff81f4c059>] ? kernel_init_freeable+0x145/0x1d0
[ÂÂÂ23.940087]ÂÂ[<ffffffff81830930>] ? rest_init+0x80/0x80
[ÂÂÂ23.940091]ÂÂ[<ffffffff81830939>] ? kernel_init+0x9/0xe0
[ÂÂÂ23.940095]ÂÂ[<ffffffff8183649f>] ? ret_from_fork+0x3f/0x70
[ÂÂÂ23.940098]ÂÂ[<ffffffff81830930>] ? rest_init+0x80/0x80
[ÂÂÂ23.940103] rcu_sched kthread starved for 19217 jiffies!
g18446744073709551443 c18446744073709551442 f0x0 s3 ->state=0x0 Â Â Â Â

[ÂÂÂ24.171981]ÂÂ3-...: (20996 ticks this GP) idle=4e3/140000000000001/0
softirq=203/203 fqs=589 
[ÂÂÂ24.181520]ÂÂÂ(t=21244 jiffies g=-173 c=-174 q=90)
[ÂÂÂ24.186877] rcu_sched kthread starved for 19463 jiffies!
g18446744073709551443 c18446744073709551442 f0x2 s3
->state=0x0ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
[ÂÂÂ24.199044] Task dump for CPU 3:
[ÂÂÂ24.202651] swapper/0ÂÂÂÂÂÂÂRÂÂrunning taskÂÂÂÂ13280ÂÂÂÂÂ1ÂÂÂÂÂÂ0
0x00000008
[ÂÂÂ24.210544]ÂÂffffffff81e3ee80 ffff88003b383dd0 ffffffff8107a616
0000000000000003
[ÂÂÂ24.218829]ÂÂffffffff81e3ee80 ffff88003b383de8 ffffffff8107c982
0000000000000004
[ÂÂÂ24.227121]ÂÂffff88003b383e18 ffffffff810a3a69 ffff88003b395dc0
ffffffff81e3ee80
[ÂÂÂ24.235414] Call Trace:
[ÂÂÂ24.238153]ÂÂ<IRQ>ÂÂ[<ffffffff8107a616>] sched_show_task+0xb6/0x110
[ÂÂÂ24.245171]ÂÂ[<ffffffff8107c982>] dump_cpu_task+0x32/0x40
[ÂÂÂ24.251213]ÂÂ[<ffffffff810a3a69>] rcu_dump_cpu_stacks+0x89/0xe0
[ÂÂÂ24.257837]ÂÂ[<ffffffff810a7379>] rcu_check_callbacks+0x4b9/0x7b0
[ÂÂÂ24.264657]ÂÂ[<ffffffff810b9f40>] ? tick_sched_do_timer+0x30/0x30
[ÂÂÂ24.271481]ÂÂ[<ffffffff810abca4>] update_process_times+0x34/0x60
[ÂÂÂ24.278203]ÂÂ[<ffffffff810b99c1>]
tick_sched_handle.isra.14+0x31/0x40
[ÂÂÂ24.285410]ÂÂ[<ffffffff810b9f7b>] tick_sched_timer+0x3b/0x70
[ÂÂÂ24.291744]ÂÂ[<ffffffff810ac49a>] __hrtimer_run_queues+0xda/0x200
[ÂÂÂ24.298563]ÂÂ[<ffffffff810acb43>] hrtimer_interrupt+0xa3/0x190
[ÂÂÂ24.305090]ÂÂ[<ffffffff81013760>] ? subcaches_show+0x40/0x40
[ÂÂÂ24.311424]ÂÂ[<ffffffff810394c3>]
local_apic_timer_interrupt+0x33/0x50
[ÂÂÂ24.318728]ÂÂ[<ffffffff81039ec8>] smp_apic_timer_interrupt+0x38/0x50
[ÂÂÂ24.325839]ÂÂ[<ffffffff81836e8f>] apic_timer_interrupt+0x7f/0x90
[ÂÂÂ24.332558]ÂÂ<EOI>ÂÂ[<ffffffff81013760>] ? subcaches_show+0x40/0x40
[ÂÂÂ24.339591]ÂÂ[<ffffffff810be900>] ?
smp_call_function_single+0xb0/0x100
[ÂÂÂ24.346993]ÂÂ[<ffffffff810be8f6>] ?
smp_call_function_single+0xa6/0x100
[ÂÂÂ24.354396]ÂÂ[<ffffffff81013760>] ? subcaches_show+0x40/0x40
[ÂÂÂ24.360728]ÂÂ[<ffffffff810147ed>] init_cache_level+0x1d/0x30
[ÂÂÂ24.367060]ÂÂ[<ffffffff81427119>] detect_cache_attributes+0x19/0x1a0
[ÂÂÂ24.374170]ÂÂ[<ffffffff814273f9>] ? cache_add_dev+0x159/0x1f0
[ÂÂÂ24.380600]ÂÂ[<ffffffff81f8cade>] ? container_dev_init+0x2d/0x2d
[ÂÂÂ24.387321]ÂÂ[<ffffffff81f8cb1e>] cacheinfo_sysfs_init+0x40/0x8a
[ÂÂÂ24.394042]ÂÂ[<ffffffff810003d6>] do_one_initcall+0xa6/0x1c0
[ÂÂÂ24.400375]ÂÂ[<ffffffff81f4c059>] kernel_init_freeable+0x145/0x1d0
[ÂÂÂ24.407290]ÂÂ[<ffffffff81830930>] ? rest_init+0x80/0x80
[ÂÂÂ24.413135]ÂÂ[<ffffffff81830939>] kernel_init+0x9/0xe0
[ÂÂÂ24.418884]ÂÂ[<ffffffff8183649f>] ret_from_fork+0x3f/0x70
[ÂÂÂ24.424923]ÂÂ[<ffffffff81830930>] ? rest_init+0x80/0x80


Kernel configuration (diff to x86_64_defconfig)

+CONFIG_E1000E=y
+CONFIG_R8169=y
+CONFIG_USB_RTL8152=y
+CONFIG_STMMAC_ETH=y
+CONFIG_STMMAC_PCI=y

-CONFIG_HPET=y
+# CONFIG_HPET is not set

-CONFIG_DRM_I915=y
+# CONFIG_DRM_I915 is not set
+CONFIG_BACKLIGHT_LCD_SUPPORT=y

-CONFIG_SND_HDA_INTEL=y
-CONFIG_SND_HDA_HWDEP=y
+# CONFIG_SND_HDA_INTEL is not set
+# CONFIG_SND_HDA_HWDEP is not set

+CONFIG_I2C_DESIGNWARE_PCI=y
+CONFIG_GPIOLIB=y
+CONFIG_GPIO_INTEL_MID=y
+CONFIG_INTEL_MID_WATCHDOG=y
+CONFIG_NOP_USB_XCEIV=y
+CONFIG_USB_CHIPIDEA=y
+CONFIG_USB_CHIPIDEA_UDC=y
+CONFIG_USB_CHIPIDEA_HOST=y
+CONFIG_X86_EXTENDED_PLATFORM=y
+CONFIG_X86_INTEL_MID=y
+CONFIG_INTEL_IDMA64=y
+CONFIG_MFD_CORE=y
+CONFIG_MFD_INTEL_LPSS_PCI=y
+CONFIG_MFD_INTEL_LPSS_ACPI=y
+CONFIG_EFI_STUB=y
+CONFIG_EFI_MIXED=y
+CONFIG_EARLY_PRINTK_EFI=y
+CONFIG_FB=y
+CONFIG_FRAMEBUFFER_CONSOLE=y
+CONFIG_DYNAMIC_DEBUG=y
+CONFIG_USB_XHCI_HCD=y
+CONFIG_USB_DWC3=y
+CONFIG_USB_DWC3_GADGET=y
+CONFIG_USB_SERIAL=y
+CONFIG_USB_SERIAL_PL2303=y
+CONFIG_USB_USBNET=y
+CONFIG_USB_NET_AX88179_178A=y
+CONFIG_USB_NET_MCS7830=y
+CONFIG_USB_NET_AX8817X=y
+CONFIG_PM_RUNTIME=y
+CONFIG_SERIAL_8250_DMA=y
+CONFIG_SERIAL_8250_PCI=y
+CONFIG_SERIAL_8250_MID=y
+CONFIG_SERIAL_8250_DW=y

-- 
Andy Shevchenko <andriy.shevchenko@xxxxxxxxx>
Intel Finland Oy

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/