Re: x86-tip tsc/tick gripage
From: Mike Galbraith
Date: Wed Apr 26 2017 - 04:17:37 EST
On Wed, 2017-04-26 at 10:02 +0200, Mike Galbraith wrote:
> tip v4.11-rc8-893-g8ec9e12aff06, trusty ole 8 socket (X7560) DL980 G7
Ew, DL980 then turned into unhappy RCU camper.
[ 316.980923] basemono: 316956000000 ts->next_tick: 316380000000 dev->next_event: 316956005002
[ 689.893122] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 689.897115] 6-...: (1456 GPs behind) idle=114/0/0 softirq=935/935 fqs=0
[ 689.897115] 7-...: (11 GPs behind) idle=358/0/0 softirq=1724/1724 fqs=0
[ 689.897115] 8-...: (11 GPs behind) idle=5ac/0/0 softirq=1618/1619 fqs=0
[ 689.897115] 9-...: (13 GPs behind) idle=318/0/0 softirq=500/500 fqs=0
[ 689.897115] 10-...: (2045 GPs behind) idle=42c/0/0 softirq=480/480 fqs=0
[ 689.897115] 12-...: (5442 GPs behind) idle=270/0/0 softirq=437/437 fqs=0
[ 689.897115] 13-...: (418 GPs behind) idle=844/0/0 softirq=449/449 fqs=0
[ 689.897115] 14-...: (14 GPs behind) idle=95c/0/0 softirq=508/509 fqs=0
[ 689.897115] 15-...: (14 GPs behind) idle=b78/0/0 softirq=475/476 fqs=0
[ 689.897115] 16-...: (9825 GPs behind) idle=22c/0/0 softirq=1308/1308 fqs=0
[ 689.897115] 17-...: (10286 GPs behind) idle=454/0/0 softirq=402/402 fqs=0
[ 689.897115] 18-...: (6734 GPs behind) idle=8e0/0/0 softirq=440/440 fqs=0
[ 689.897115] 19-...: (6734 GPs behind) idle=604/0/0 softirq=378/378 fqs=0
[ 689.897115] 22-...: (5162 GPs behind) idle=3c8/0/0 softirq=391/391 fqs=0
[ 689.897115] 24-...: (229 GPs behind) idle=3d0/0/0 softirq=1470/1470 fqs=0
[ 689.897115] 25-...: (9809 GPs behind) idle=2f0/0/0 softirq=436/436 fqs=0
[ 689.897115] 26-...: (9809 GPs behind) idle=418/0/0 softirq=413/413 fqs=0
[ 689.897115] 27-...: (9809 GPs behind) idle=458/0/0 softirq=438/438 fqs=0
[ 689.897115] 28-...: (9809 GPs behind) idle=dd4/0/0 softirq=494/494 fqs=0
[ 689.897115] 29-...: (9572 GPs behind) idle=3fc/0/0 softirq=400/400 fqs=0
[ 689.897115] 30-...: (4950 GPs behind) idle=084/0/0 softirq=422/422 fqs=0
[ 689.897115] 31-...: (4993 GPs behind) idle=ba4/0/0 softirq=366/366 fqs=0
[ 689.897115] 32-...: (2048 GPs behind) idle=608/0/0 softirq=1258/1259 fqs=0
[ 689.897115] 33-...: (2046 GPs behind) idle=f14/0/0 softirq=1267/1268 fqs=0
[ 689.897115] 34-...: (0 ticks this GP) idle=950/0/0 softirq=357/357 fqs=0
[ 689.897115] 35-...: (9813 GPs behind) idle=7a8/0/0 softirq=301/301 fqs=0
[ 689.897115] 36-...: (9813 GPs behind) idle=5c0/0/0 softirq=299/299 fqs=0
[ 689.897115] 37-...: (9828 GPs behind) idle=478/0/0 softirq=286/286 fqs=0
[ 689.897115] 38-...: (0 ticks this GP) idle=8d4/0/0 softirq=307/307 fqs=0
[ 689.897115] 39-...: (2048 GPs behind) idle=2ac/0/0 softirq=304/305 fqs=0
[ 689.897115] 40-...: (0 ticks this GP) idle=bd0/0/0 softirq=1184/1184 fqs=0
[ 689.897115] 41-...: (428 GPs behind) idle=d14/0/0 softirq=339/339 fqs=0
[ 689.897115] 42-...: (243 GPs behind) idle=c38/0/0 softirq=338/338 fqs=0
[ 689.897115] 43-...: (9809 GPs behind) idle=870/0/0 softirq=289/289 fqs=0
[ 689.897115] 46-...: (9815 GPs behind) idle=654/0/0 softirq=261/261 fqs=0
[ 689.897115] 47-...: (2134 GPs behind) idle=c98/0/0 softirq=303/303 fqs=0
[ 689.897115] 48-...: (571 GPs behind) idle=4c0/0/0 softirq=1135/1135 fqs=0
[ 689.897115] 49-...: (2211 GPs behind) idle=bcc/0/0 softirq=267/267 fqs=0
[ 689.897115] 50-...: (2195 GPs behind) idle=334/0/0 softirq=232/233 fqs=0
[ 689.897115] 51-...: (97 GPs behind) idle=e30/0/0 softirq=374/374 fqs=0
[ 689.897115] 52-...: (252 GPs behind) idle=c14/0/0 softirq=207/208 fqs=0
[ 689.897115] 53-...: (0 ticks this GP) idle=dd4/0/0 softirq=218/218 fqs=0
[ 689.897115] 54-...: (526 GPs behind) idle=ce8/0/0 softirq=292/293 fqs=0
[ 689.897115] 55-...: (573 GPs behind) idle=924/0/0 softirq=191/192 fqs=0
[ 689.897115] 56-...: (68 GPs behind) idle=028/0/0 softirq=1141/1141 fqs=0
[ 689.897115] 57-...: (5442 GPs behind) idle=f48/0/0 softirq=319/319 fqs=0
[ 689.897115] 58-...: (5211 GPs behind) idle=18c/0/0 softirq=185/185 fqs=0
[ 689.897115] 59-...: (255 GPs behind) idle=a30/0/0 softirq=276/277 fqs=0
[ 689.897115] 60-...: (5442 GPs behind) idle=e54/0/0 softirq=155/155 fqs=0
[ 689.897115] 61-...: (5151 GPs behind) idle=270/0/0 softirq=169/170 fqs=0
[ 689.897115] 62-...: (9819 GPs behind) idle=c50/0/0 softirq=116/116 fqs=0
[ 689.897115] 63-...: (2195 GPs behind) idle=e4c/0/0 softirq=141/142 fqs=0
[ 689.897115] (detected by 2, t=15689 jiffies, g=12275, c=12274, q=520)
[ 689.897115] Sending NMI from CPU 2 to CPUs 6:
[ 693.749331] NMI backtrace for cpu 6 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 7:
[ 693.750354] NMI backtrace for cpu 7 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 8:
[ 693.751358] NMI backtrace for cpu 8 skipped: idling at pc 0xffffffff8167bdc8
[ 689.897115] Sending NMI from CPU 2 to CPUs 9:
[ 693.752357] NMI backtrace for cpu 9 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 10:
[ 693.753360] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 12:
[ 693.754358] NMI backtrace for cpu 12 skipped: idling at pc 0xffffffff8167bdcd
[ 689.897115] Sending NMI from CPU 2 to CPUs 13:
[ 693.755360] NMI backtrace for cpu 13 skipped: idling at pc 0xffffffff8167bdd0
[ 689.897115] Sending NMI from CPU 2 to CPUs 14:
[ 693.756361] NMI backtrace for cpu 14 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 15:
[ 693.757364] NMI backtrace for cpu 15 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 16:
[ 693.758561] NMI backtrace for cpu 16 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 17:
[ 693.759368] NMI backtrace for cpu 17 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 18:
[ 693.760370] NMI backtrace for cpu 18 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 19:
[ 693.761377] NMI backtrace for cpu 19 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 22:
[ 693.762377] NMI backtrace for cpu 22 skipped: idling at pc 0xffffffff8167bdd0
[ 689.897115] Sending NMI from CPU 2 to CPUs 24:
[ 693.763390] NMI backtrace for cpu 24 skipped: idling at pc 0xffffffff8167bdcd
[ 689.897115] Sending NMI from CPU 2 to CPUs 25:
[ 693.764381] NMI backtrace for cpu 25 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 26:
[ 693.765390] NMI backtrace for cpu 26 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 27:
[ 693.766389] NMI backtrace for cpu 27 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 28:
[ 693.767390] NMI backtrace for cpu 28 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 29:
[ 693.768392] NMI backtrace for cpu 29 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 30:
[ 693.769396] NMI backtrace for cpu 30 skipped: idling at pc 0xffffffff8167bdc8
[ 689.897115] Sending NMI from CPU 2 to CPUs 31:
[ 689.897115] NMI backtrace for cpu 31
[ 689.897115] CPU: 31 PID: 176 Comm: rcuos/24 Tainted: G W E 4.11.0-tip-default #57
[ 689.897115] Hardware name: Hewlett-Packard ProLiant DL980 G7, BIOS P66 07/07/2010
[ 689.897115] task: ffff88017dac1540 task.stack: ffffc900013dc000
[ 689.897115] RIP: 0010:native_queued_spin_lock_slowpath+0x170/0x1a0
[ 689.897115] RSP: 0018:ffffc900013dfe88 EFLAGS: 00000002
[ 689.897115] RAX: 0000000000780101 RBX: 0000000000000202 RCX: 0000000000000001
[ 689.897115] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffffffff81caac40
[ 689.897115] RBP: ffff88027741c600 R08: 0000000000000101 R09: 000000000000025d
[ 689.897115] R10: 0000000000000265 R11: 0000000000000005 R12: 0000000000000001
[ 689.897115] R13: ffff88017dac1540 R14: 0000000000000000 R15: ffff88017dac1540
[ 689.897115] FS: 0000000000000000(0000) GS:ffff8802775c0000(0000) knlGS:0000000000000000
[ 689.897115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 689.897115] CR2: 00007f30fd0ce000 CR3: 0000000001c09000 CR4: 00000000000006e0
[ 689.897115] Call Trace:
[ 689.897115] ? queued_spin_lock_slowpath+0x7/0xa
[ 689.897115] ? _raw_spin_lock_irqsave+0x39/0x40
[ 689.897115] ? rcu_nocb_kthread+0x24f/0x480
[ 689.897115] ? kthread+0x107/0x140
[ 689.897115] ? rcu_gp_kthread+0x8c0/0x8c0
[ 689.897115] ? kthread_park+0x60/0x60
[ 689.897115] ? ret_from_fork+0x26/0x40
[ 689.897115] Code: 89 d0 66 31 c0 39 c1 74 e7 4d 85 c9 c6 07 01 74 2d 41 c7 41 08 01 00 00 00 e9 52 ff ff ff 83 fa 01 74 17 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 f3 c3 f3 90 4d 8b
[ 689.897115] Sending NMI from CPU 2 to CPUs 32:
[ 693.771432] NMI backtrace for cpu 32 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 33:
[ 693.772426] NMI backtrace for cpu 33 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 34:
[ 693.773431] NMI backtrace for cpu 34 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 35:
[ 693.774432] NMI backtrace for cpu 35 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 36:
[ 693.775434] NMI backtrace for cpu 36 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 37:
[ 693.776436] NMI backtrace for cpu 37 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 38:
[ 693.777442] NMI backtrace for cpu 38 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 39:
[ 693.778444] NMI backtrace for cpu 39 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 40:
[ 693.779456] NMI backtrace for cpu 40 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 41:
[ 693.780448] NMI backtrace for cpu 41 skipped: idling at pc 0xffffffff8167bdd0
[ 689.897115] Sending NMI from CPU 2 to CPUs 42:
[ 693.781453] NMI backtrace for cpu 42 skipped: idling at pc 0xffffffff8167bdca
[ 689.897115] Sending NMI from CPU 2 to CPUs 43:
[ 693.782457] NMI backtrace for cpu 43 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 46:
[ 693.783459] NMI backtrace for cpu 46 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 47:
[ 693.784463] NMI backtrace for cpu 47 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 48:
[ 693.785477] NMI backtrace for cpu 48 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 49:
[ 693.786470] NMI backtrace for cpu 49 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 50:
[ 693.787471] NMI backtrace for cpu 50 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 51:
[ 693.788475] NMI backtrace for cpu 51 skipped: idling at pc 0xffffffff8167bdcd
[ 689.897115] Sending NMI from CPU 2 to CPUs 52:
[ 693.789481] NMI backtrace for cpu 52 skipped: idling at pc 0xffffffff8167bdca
[ 689.897115] Sending NMI from CPU 2 to CPUs 53:
[ 693.790482] NMI backtrace for cpu 53 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 54:
[ 693.791585] NMI backtrace for cpu 54 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 55:
[ 693.792587] NMI backtrace for cpu 55 skipped: idling at pc 0xffffffff8167bdca
[ 689.897115] Sending NMI from CPU 2 to CPUs 56:
[ 693.793606] NMI backtrace for cpu 56 skipped: idling at pc 0xffffffff8167bdca
[ 689.897115] Sending NMI from CPU 2 to CPUs 57:
[ 693.794593] NMI backtrace for cpu 57 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 58:
[ 693.795595] NMI backtrace for cpu 58 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 59:
[ 693.796599] NMI backtrace for cpu 59 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 60:
[ 693.797605] NMI backtrace for cpu 60 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 61:
[ 693.798605] NMI backtrace for cpu 61 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 62:
[ 693.799611] NMI backtrace for cpu 62 skipped: idling at pc 0xffffffff8167bcc6
[ 689.897115] Sending NMI from CPU 2 to CPUs 63:
[ 693.800610] NMI backtrace for cpu 63 skipped: idling at pc 0xffffffff8167bdcd
[ 1447.897093] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1447.901088] 5-...: (0 ticks this GP) idle=a6c/0/0 softirq=840/840 fqs=0
[ 1447.901088] 7-...: (46 GPs behind) idle=4f8/0/0 softirq=1841/1841 fqs=0
[ 1447.901088] 8-...: (58 GPs behind) idle=ab8/0/0 softirq=1994/1995 fqs=0
[ 1447.901088] 9-...: (58 GPs behind) idle=ebc/0/0 softirq=540/540 fqs=0
[ 1447.901088] 10-...: (1681 GPs behind) idle=c14/0/0 softirq=488/489 fqs=0
[ 1447.901088] 11-...: (0 ticks this GP) idle=dd8/0/0 softirq=1363/1363 fqs=0
[ 1447.901088] 12-...: (17237 GPs behind) idle=a34/0/0 softirq=437/438 fqs=0
[ 1447.901088] 13-...: (12213 GPs behind) idle=004/0/0 softirq=449/450 fqs=0
[ 1447.901088] 14-...: (58 GPs behind) idle=eec/0/0 softirq=704/705 fqs=0
[ 1447.901088] 15-...: (58 GPs behind) idle=0f8/0/0 softirq=3031/3031 fqs=0
[ 1447.901088] 16-...: (1374 GPs behind) idle=3a0/0/0 softirq=1310/1310 fqs=0
[ 1447.901088] 17-...: (1032 GPs behind) idle=ac4/0/0 softirq=412/412 fqs=0
[ 1447.901088] 18-...: (847 GPs behind) idle=d00/0/0 softirq=449/449 fqs=0
[ 1447.901088] 19-...: (1374 GPs behind) idle=8ec/0/0 softirq=381/381 fqs=0
[ 1447.901088] 20-...: (960 GPs behind) idle=2e4/0/0 softirq=407/407 fqs=0
[ 1447.901088] 21-...: (900 GPs behind) idle=6e8/0/0 softirq=463/463 fqs=0
[ 1447.901088] 22-...: (1036 GPs behind) idle=a70/0/0 softirq=399/399 fqs=0
[ 1447.901088] 23-...: (1373 GPs behind) idle=f08/0/0 softirq=377/377 fqs=0
[ 1447.901088] 24-...: (6627 GPs behind) idle=3d0/0/0 softirq=1492/1492 fqs=0
[ 1447.901088] 25-...: (11795 GPs behind) idle=2ec/0/0 softirq=438/438 fqs=0
[ 1447.901088] 26-...: (11795 GPs behind) idle=4c4/0/0 softirq=415/415 fqs=0
[ 1447.901088] 27-...: (11795 GPs behind) idle=4d0/0/0 softirq=440/440 fqs=0
[ 1447.901088] 28-...: (21604 GPs behind) idle=dec/0/0 softirq=494/495 fqs=0
[ 1447.901088] 29-...: (21367 GPs behind) idle=394/0/0 softirq=400/401 fqs=0
[ 1447.901088] 30-...: (16745 GPs behind) idle=0f0/0/0 softirq=422/423 fqs=0
[ 1447.901088] 31-...: (11795 GPs behind) idle=b94/0/0 softirq=367/367 fqs=0
[ 1447.901088] 32-...: (2042 GPs behind) idle=fa0/0/0 softirq=1263/1264 fqs=0
[ 1447.901088] 33-...: (120 GPs behind) idle=d88/0/0 softirq=1283/1284 fqs=0
[ 1447.901088] 34-...: (123 GPs behind) idle=4d8/0/0 softirq=359/359 fqs=0
[ 1447.901088] 35-...: (21608 GPs behind) idle=184/0/0 softirq=301/302 fqs=0
[ 1447.901088] 36-...: (21608 GPs behind) idle=fa0/0/0 softirq=299/300 fqs=0
[ 1447.901088] 37-...: (21623 GPs behind) idle=f0c/0/0 softirq=286/287 fqs=0
[ 1447.901088] 38-...: (214 GPs behind) idle=e54/0/0 softirq=324/324 fqs=0
[ 1447.901088] 39-...: (84 GPs behind) idle=c74/0/0 softirq=316/316 fqs=0
[ 1447.901088] 40-...: (0 ticks this GP) idle=0b0/0/0 softirq=1201/1201 fqs=0
[ 1447.901088] 42-...: (12038 GPs behind) idle=76c/0/0 softirq=338/339 fqs=0
[ 1447.901088] 43-...: (11795 GPs behind) idle=404/0/0 softirq=290/290 fqs=0
[ 1447.901088] 45-...: (462 GPs behind) idle=e90/0/0 softirq=336/336 fqs=0
[ 1447.901088] 46-...: (266 GPs behind) idle=de8/0/0 softirq=269/269 fqs=0
[ 1447.901088] 47-...: (161 GPs behind) idle=20c/0/0 softirq=309/309 fqs=0
[ 1447.901088] 48-...: (11795 GPs behind) idle=12c/0/0 softirq=1137/1138 fqs=0
[ 1447.901088] 49-...: (14006 GPs behind) idle=7a0/0/0 softirq=267/268 fqs=0
[ 1447.901088] 50-...: (13990 GPs behind) idle=f4c/0/0 softirq=232/234 fqs=0
[ 1447.901088] 52-...: (0 ticks this GP) idle=7bc/0/0 softirq=213/213 fqs=0
[ 1447.901088] 53-...: (63 GPs behind) idle=df0/0/0 softirq=238/238 fqs=0
[ 1447.901088] 54-...: (5326 GPs behind) idle=01c/0/0 softirq=300/300 fqs=0
[ 1447.901088] 55-...: (12368 GPs behind) idle=460/0/0 softirq=191/193 fqs=0
[ 1447.901088] 57-...: (17237 GPs behind) idle=ed8/0/0 softirq=319/320 fqs=0
[ 1447.901088] 58-...: (7471 GPs behind) idle=f18/0/0 softirq=190/190 fqs=0
[ 1447.901088] 59-...: (7474 GPs behind) idle=c70/0/0 softirq=279/280 fqs=0
[ 1447.901088] 60-...: (11788 GPs behind) idle=96c/0/0 softirq=157/158 fqs=0
[ 1447.901088] 61-...: (759 GPs behind) idle=73c/0/0 softirq=178/178 fqs=0
[ 1447.901088] 62-...: (11795 GPs behind) idle=b7c/0/0 softirq=117/117 fqs=0
[ 1447.901088] 63-...: (11795 GPs behind) idle=d8c/0/0 softirq=143/143 fqs=0
[ 1447.901088] (detected by 2, t=15938 jiffies, g=24070, c=24069, q=1057)
[ 1447.901088] Sending NMI from CPU 2 to CPUs 5:
[ 1451.999285] NMI backtrace for cpu 5 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 7:
[ 1452.000294] NMI backtrace for cpu 7 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 8:
[ 1452.001297] NMI backtrace for cpu 8 skipped: idling at pc 0xffffffff8167bdcd
[ 1447.901088] Sending NMI from CPU 2 to CPUs 9:
[ 1452.002296] NMI backtrace for cpu 9 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 10:
[ 1452.003298] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 11:
[ 1452.004347] NMI backtrace for cpu 11 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 12:
[ 1452.005301] NMI backtrace for cpu 12 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 13:
[ 1452.006300] NMI backtrace for cpu 13 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 14:
[ 1452.007301] NMI backtrace for cpu 14 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 15:
[ 1452.008303] NMI backtrace for cpu 15 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 16:
[ 1452.009308] NMI backtrace for cpu 16 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 17:
[ 1452.010307] NMI backtrace for cpu 17 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 18:
[ 1452.011309] NMI backtrace for cpu 18 skipped: idling at pc 0xffffffff8167bdc8
[ 1447.901088] Sending NMI from CPU 2 to CPUs 19:
[ 1452.012313] NMI backtrace for cpu 19 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 20:
[ 1452.013319] NMI backtrace for cpu 20 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 21:
[ 1452.014316] NMI backtrace for cpu 21 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 22:
[ 1452.015318] NMI backtrace for cpu 22 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 23:
[ 1452.016320] NMI backtrace for cpu 23 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 24:
[ 1452.017332] NMI backtrace for cpu 24 skipped: idling at pc 0xffffffff8167bdcd
[ 1447.901088] Sending NMI from CPU 2 to CPUs 25:
[ 1452.018325] NMI backtrace for cpu 25 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 26:
[ 1452.019329] NMI backtrace for cpu 26 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 27:
[ 1452.020330] NMI backtrace for cpu 27 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 28:
[ 1452.021335] NMI backtrace for cpu 28 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 29:
[ 1452.022334] NMI backtrace for cpu 29 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 30:
[ 1452.023335] NMI backtrace for cpu 30 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 31:
[ 1452.024339] NMI backtrace for cpu 31 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 32:
[ 1452.025347] NMI backtrace for cpu 32 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 33:
[ 1452.026346] NMI backtrace for cpu 33 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 34:
[ 1452.027346] NMI backtrace for cpu 34 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 35:
[ 1452.028348] NMI backtrace for cpu 35 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 36:
[ 1452.029356] NMI backtrace for cpu 36 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 37:
[ 1452.030354] NMI backtrace for cpu 37 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 38:
[ 1452.031358] NMI backtrace for cpu 38 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 39:
[ 1452.032359] NMI backtrace for cpu 39 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 40:
[ 1452.033368] NMI backtrace for cpu 40 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 42:
[ 1452.034365] NMI backtrace for cpu 42 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 43:
[ 1452.035367] NMI backtrace for cpu 43 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 45:
[ 1452.036371] NMI backtrace for cpu 45 skipped: idling at pc 0xffffffff8167bdc8
[ 1447.901088] Sending NMI from CPU 2 to CPUs 46:
[ 1452.037374] NMI backtrace for cpu 46 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 47:
[ 1452.038476] NMI backtrace for cpu 47 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 48:
[ 1452.039483] NMI backtrace for cpu 48 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 49:
[ 1452.040484] NMI backtrace for cpu 49 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 50:
[ 1452.041486] NMI backtrace for cpu 50 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 52:
[ 1452.042488] NMI backtrace for cpu 52 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 53:
[ 1452.043489] NMI backtrace for cpu 53 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 54:
[ 1452.044493] NMI backtrace for cpu 54 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 55:
[ 1452.045500] NMI backtrace for cpu 55 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 57:
[ 1452.046499] NMI backtrace for cpu 57 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 58:
[ 1452.047502] NMI backtrace for cpu 58 skipped: idling at pc 0xffffffff8167bdca
[ 1447.901088] Sending NMI from CPU 2 to CPUs 59:
[ 1452.048504] NMI backtrace for cpu 59 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 60:
[ 1452.049511] NMI backtrace for cpu 60 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 61:
[ 1452.050509] NMI backtrace for cpu 61 skipped: idling at pc 0xffffffff8167bdcd
[ 1447.901088] Sending NMI from CPU 2 to CPUs 62:
[ 1452.051510] NMI backtrace for cpu 62 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] Sending NMI from CPU 2 to CPUs 63:
[ 1452.052514] NMI backtrace for cpu 63 skipped: idling at pc 0xffffffff8167bcc6
[ 1447.901088] rcu_sched kthread starved for 16040 jiffies! g24070 c24069 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[ 1447.901088] rcu_sched S 0 9 2 0x00000000
[ 1447.901088] Call Trace:
[ 1447.901088] ? __schedule+0x28f/0x8b0
[ 1447.901088] ? schedule+0x39/0x90
[ 1447.901088] ? schedule_timeout+0x180/0x2f0
[ 1447.901088] ? call_timer_fn+0x150/0x150
[ 1447.901088] ? rcu_gp_kthread+0x3b4/0x8c0
[ 1447.901088] ? kthread+0x107/0x140
[ 1447.901088] ? force_qs_rnp+0x1a0/0x1a0
[ 1447.901088] ? kthread_park+0x60/0x60
[ 1447.901088] ? ret_from_fork+0x26/0x40