Re: RCU stall warnings...

[Paul E. McKenney]

On Mon, Jul 24, 2017 at 03:32:48PM -0700, David Miller wrote:
> 
> Paul and other RCU experts,
> 
> Starting with 4.13-rc1 we're getting RCU stall dumps on sparc64, they
> definitely didn't happen in 4.12
> 
> I tried to look for low hanging fruit in the kernel/rcu/ changes this
> merge window, but nothing really stood out for me.
> 
> This only seems to trigger when the system is mostly idle.
> 
> Kconfig wise PREEMPT is disabled, and NO_HZ_COMMON and NO_HOZ_IDLE are
> both enabled.
> 
> The dumps always look the same, see below.  The idle values are even,
> so those cpus should be in dynticks mode.
> 
> Perhaps I'm not doing something properly in our arch idle loop, but I
> don't see anything obvious there.

It looks like the system isn't letting the rcu_sched grace-period kthread
run:

[402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1

This kthread tried to wait for a few jiffies (the exact number depends
on HZ and the number of CPUs), but 2,757 jiffies have elapsed and it is
still waiting.  This kthread is responsible for detecting idle CPUs and
reporting quiescent states on their behalf, so if this kthread doesn't
get a chance to run, then the stall warnings you are seeing are expected
behavior.

I am seeing someething like sort of like this in my rcutorture runs,
but only when I boot with nr_cpus quite a bit bigger than maxcpus, as in
something like nr_cpus=43 and maxcpus=8.  This causes 8 CPUs to be brought
online at the usual time, and the other 35 come online some time later.
One difference from your situation is that I see the grace-period
kthread in ->state=0x401 (TASK_WAKING) instead of your ->state=0x1.
If I send extra wakeups to the grace-period kthread (which shouldn't be
needed), it does make progress, but then other kthreads fall into that
same half-woken state.

So now that I ahve shared the full extent of my ignorance on this topic,
any ideas?  ;-)

							Thanx, Paul

> [402131.682905] INFO: rcu_sched detected stalls on CPUs/tasks:
> [402131.693768]         2-...: (4304 GPs behind) idle=990/0/0 softirq=80469/80469 fqs=0 
> [402131.708188]         3-...: (3286 GPs behind) idle=0bc/0/0 softirq=77239/77241 fqs=0 
> [402131.722595]         4-...: (5513 GPs behind) idle=2b4/0/0 softirq=74384/74386 fqs=0 
> [402131.737003]         5-...: (3596 GPs behind) idle=ad0/0/0 softirq=73109/73109 fqs=0 
> [402131.751411]         6-...: (3592 GPs behind) idle=e78/0/0 softirq=67679/67679 fqs=0 
> [402131.765835]         7-...: (3591 GPs behind) idle=9ac/0/0 softirq=65887/65889 fqs=0 
> [402131.780243]         9-...: (5 GPs behind) idle=724/0/0 softirq=88433/88436 fqs=0 
> [402131.794121]         11-...: (3286 GPs behind) idle=be4/0/0 softirq=81587/81589 fqs=0 
> [402131.808724]         12-...: (3596 GPs behind) idle=ed0/0/0 softirq=79542/79544 fqs=0 
> [402131.823307]         13-...: (3596 GPs behind) idle=9f8/0/0 softirq=73652/73655 fqs=0 
> [402131.837891]         14-...: (3593 GPs behind) idle=d3c/0/0 softirq=72326/72328 fqs=0 
> [402131.852476]         15-...: (3596 GPs behind) idle=6f4/0/0 softirq=69730/69732 fqs=0 
> [402131.867060]         20-...: (2566 GPs behind) idle=b80/0/0 softirq=79921/79925 fqs=0 
> [402131.881644]         21-...: (4701 GPs behind) idle=944/0/0 softirq=73890/73890 fqs=0 
> [402131.896228]         22-...: (2563 GPs behind) idle=b58/0/0 softirq=74554/74554 fqs=0 
> [402131.910812]         23-...: (3285 GPs behind) idle=b08/0/0 softirq=112972/112972 fqs=0 
> [402131.925749]         27-...: (3596 GPs behind) idle=5f4/0/0 softirq=79524/79524 fqs=0 
> [402131.940333]         28-...: (4070 GPs behind) idle=b30/0/0 softirq=71061/71063 fqs=0 
> [402131.954917]         29-...: (2430 GPs behind) idle=4b4/0/0 softirq=72431/72433 fqs=0 
> [402131.969501]         30-...: (389 GPs behind) idle=65c/0/0 softirq=68508/68510 fqs=0 
> [402131.983925]         31-...: (170 GPs behind) idle=174/0/0 softirq=72802/72804 fqs=0 
> [402131.998334]         33-...: (1627 GPs behind) idle=d2c/0/0 softirq=89821/89823 fqs=0 
> [402132.012918]         34-...: (391 GPs behind) idle=8b0/0/0 softirq=82561/82563 fqs=0 
> [402132.027326]         35-...: (169 GPs behind) idle=700/0/0 softirq=77960/77962 fqs=0 
> [402132.041734]         36-...: (3590 GPs behind) idle=cb4/0/0 softirq=79230/79232 fqs=0 
> [402132.056334]         37-...: (391 GPs behind) idle=264/0/0 softirq=71121/71123 fqs=0 
> [402132.070743]         38-...: (3594 GPs behind) idle=604/0/0 softirq=71422/71422 fqs=0 
> [402132.085327]         39-...: (3409 GPs behind) idle=eb4/0/0 softirq=67121/67121 fqs=0 
> [402132.099911]         42-...: (172 GPs behind) idle=d4c/0/0 softirq=121584/121588 fqs=0 
> [402132.114671]         43-...: (388 GPs behind) idle=3f0/0/0 softirq=80541/80541 fqs=0 
> [402132.129079]         44-...: (3605 GPs behind) idle=410/0/0 softirq=73502/73502 fqs=0 
> [402132.143663]         45-...: (3591 GPs behind) idle=078/0/0 softirq=74292/74292 fqs=0 
> [402132.158247]         46-...: (3594 GPs behind) idle=800/0/0 softirq=73200/73200 fqs=0 
> [402132.172832]         47-...: (3599 GPs behind) idle=460/0/0 softirq=68045/68047 fqs=0 
> [402132.187433]         50-...: (97 GPs behind) idle=180/0/0 softirq=81843/81843 fqs=0 
> [402132.201662]         51-...: (172 GPs behind) idle=5bc/0/0 softirq=76273/76277 fqs=0 
> [402132.216072]         52-...: (2915 GPs behind) idle=688/0/0 softirq=76044/76046 fqs=0 
> [402132.230656]         53-...: (2564 GPs behind) idle=6dc/0/0 softirq=75980/75982 fqs=0 
> [402132.245240]         54-...: (6357 GPs behind) idle=a40/0/0 softirq=72584/72584 fqs=0 
> [402132.259841]         55-...: (3286 GPs behind) idle=c34/0/0 softirq=70795/70795 fqs=0 
> [402132.274425]         57-...: (4 GPs behind) idle=d0c/0/0 softirq=90478/90480 fqs=0 
> [402132.288479]         59-...: (0 ticks this GP) idle=d2c/0/0 softirq=79767/79767 fqs=0 
> [402132.303065]         61-...: (5511 GPs behind) idle=4f0/0/0 softirq=78603/78603 fqs=0 
> [402132.317649]         62-...: (5509 GPs behind) idle=e88/0/0 softirq=73573/73575 fqs=0 
> [402132.332233]         63-...: (5518 GPs behind) idle=d94/0/0 softirq=71343/71343 fqs=0 
> [402132.346832]         64-...: (55 GPs behind) idle=a14/0/0 softirq=122673/122673 fqs=0 
> [402132.361417]         65-...: (134 GPs behind) idle=4a8/0/0 softirq=97047/97047 fqs=0 
> [402132.375825]         66-...: (2523 GPs behind) idle=418/0/0 softirq=81713/81713 fqs=0 
> [402132.390409]         67-...: (2470 GPs behind) idle=000/0/0 softirq=84375/84377 fqs=0 
> [402132.405001]         68-...: (2565 GPs behind) idle=fcc/0/0 softirq=72096/72096 fqs=0 
> [402132.419586]         69-...: (3655 GPs behind) idle=1e4/0/0 softirq=71565/71567 fqs=0 
> [402132.434170]         70-...: (3597 GPs behind) idle=89c/0/0 softirq=68341/68341 fqs=0 
> [402132.448754]         71-...: (3605 GPs behind) idle=f94/0/0 softirq=64666/64668 fqs=0 
> [402132.463333]         72-...: (49 GPs behind) idle=b40/0/0 softirq=112848/112848 fqs=0 
> [402132.477924]         73-...: (160 GPs behind) idle=db8/0/0 softirq=85959/85961 fqs=0 
> [402132.492340]         74-...: (157 GPs behind) idle=d28/0/0 softirq=81416/81416 fqs=0 
> [402132.506748]         75-...: (186 GPs behind) idle=34c/0/0 softirq=85910/85912 fqs=0 
> [402132.521158]         76-...: (4072 GPs behind) idle=764/0/0 softirq=75200/75202 fqs=0 
> [402132.535744]         77-...: (3695 GPs behind) idle=e28/0/0 softirq=76678/76678 fqs=0 
> [402132.550331]         78-...: (3691 GPs behind) idle=5b4/0/0 softirq=72681/72681 fqs=0 
> [402132.564925]         79-...: (11143 GPs behind) idle=4a8/0/0 softirq=73212/73212 fqs=0 
> [402132.579670]         80-...: (132 GPs behind) idle=044/0/0 softirq=117329/117329 fqs=0 
> [402132.594429]         81-...: (157 GPs behind) idle=cb4/0/0 softirq=87921/87923 fqs=0 
> [402132.608846]         82-...: (2432 GPs behind) idle=c3c/0/0 softirq=83531/83531 fqs=0 
> [402132.623432]         83-...: (389 GPs behind) idle=c94/0/0 softirq=79845/79847 fqs=0 
> [402132.637840]         84-...: (3588 GPs behind) idle=990/0/0 softirq=75763/75765 fqs=0 
> [402132.652424]         85-...: (3593 GPs behind) idle=be0/0/0 softirq=76481/76483 fqs=0 
> [402132.667020]         86-...: (5254 GPs behind) idle=4c0/0/0 softirq=73565/73567 fqs=0 
> [402132.681605]         87-...: (5507 GPs behind) idle=4a8/0/0 softirq=70595/70597 fqs=0 
> [402132.696182]         88-...: (162 GPs behind) idle=2ac/0/0 softirq=120362/120364 fqs=0 
> [402132.710942]         89-...: (155 GPs behind) idle=f6c/0/0 softirq=86298/86300 fqs=0 
> [402132.725367]         90-...: (161 GPs behind) idle=23c/0/0 softirq=81845/81845 fqs=0 
> [402132.739779]         91-...: (5236 GPs behind) idle=a84/0/0 softirq=76524/76524 fqs=0 
> [402132.754363]         92-...: (5073 GPs behind) idle=46c/0/0 softirq=78364/78366 fqs=0 
> [402132.768952]         93-...: (7496 GPs behind) idle=590/0/0 softirq=75926/75928 fqs=0 
> [402132.783536]         94-...: (3594 GPs behind) idle=fdc/0/0 softirq=73619/73619 fqs=0 
> [402132.798110]         95-...: (446 GPs behind) idle=4d8/0/0 softirq=67953/67953 fqs=0 
> [402132.812535]         96-...: (156 GPs behind) idle=4b0/0/0 softirq=151231/151231 fqs=0 
> [402132.827295]         97-...: (2066 GPs behind) idle=4a0/0/0 softirq=90382/90384 fqs=0 
> [402132.841879]         98-...: (3592 GPs behind) idle=e10/0/0 softirq=85418/85420 fqs=0 
> [402132.856454]         99-...: (90 GPs behind) idle=6c4/0/0 softirq=116865/116865 fqs=0 
> [402132.871038]         100-...: (162 GPs behind) idle=aa0/0/0 softirq=77088/77089 fqs=0 
> [402132.885627]         101-...: (157 GPs behind) idle=5e8/0/0 softirq=75034/75036 fqs=0 
> [402132.900211]         102-...: (3591 GPs behind) idle=a08/0/0 softirq=72174/72176 fqs=0 
> [402132.914971]         103-...: (3605 GPs behind) idle=f2c/0/0 softirq=68188/68190 fqs=0 
> [402132.929736]         104-...: (155 GPs behind) idle=0cc/0/0 softirq=107026/107026 fqs=0 
> [402132.944672]         105-...: (157 GPs behind) idle=c24/0/0 softirq=83549/83551 fqs=0 
> [402132.959256]         106-...: (1625 GPs behind) idle=6dc/0/0 softirq=80774/80774 fqs=0 
> [402132.974007]         107-...: (3589 GPs behind) idle=d4c/0/0 softirq=82554/82556 fqs=0 
> [402132.988768]         108-...: (3588 GPs behind) idle=cc8/0/0 softirq=76988/76990 fqs=0 
> [402133.003528]         109-...: (3597 GPs behind) idle=644/0/0 softirq=74768/74768 fqs=0 
> [402133.018288]         110-...: (5176 GPs behind) idle=96c/0/0 softirq=72869/72871 fqs=0 
> [402133.033048]         111-...: (3401 GPs behind) idle=458/0/0 softirq=70859/70859 fqs=0 
> [402133.047809]         112-...: (125 GPs behind) idle=308/0/0 softirq=108176/108176 fqs=0 
> [402133.062745]         113-...: (157 GPs behind) idle=e24/0/0 softirq=83982/83984 fqs=0 
> [402133.077321]         114-...: (725 GPs behind) idle=4b0/0/0 softirq=80102/80103 fqs=0 
> [402133.091922]         115-...: (3596 GPs behind) idle=744/0/0 softirq=79547/79547 fqs=0 
> [402133.106675]         116-...: (4977 GPs behind) idle=298/0/0 softirq=75316/75318 fqs=0 
> [402133.121435]         117-...: (3591 GPs behind) idle=00c/0/0 softirq=75114/75114 fqs=0 
> [402133.136188]         118-...: (3592 GPs behind) idle=d70/0/0 softirq=73200/73200 fqs=0 
> [402133.150946]         119-...: (5255 GPs behind) idle=d28/0/0 softirq=67479/67479 fqs=0 
> [402133.165708]         120-...: (138 GPs behind) idle=e08/0/0 softirq=112097/112099 fqs=0 
> [402133.180641]         121-...: (156 GPs behind) idle=4d0/0/0 softirq=89608/89608 fqs=0 
> [402133.195225]         122-...: (389 GPs behind) idle=648/0/0 softirq=81157/81159 fqs=0 
> [402133.209814]         123-...: (389 GPs behind) idle=3b0/0/0 softirq=81959/81961 fqs=0 
> [402133.224405]         124-...: (4102 GPs behind) idle=b50/0/0 softirq=75522/75524 fqs=0 
> [402133.239165]         125-...: (3591 GPs behind) idle=f64/0/0 softirq=74702/74702 fqs=0 
> [402133.253926]         126-...: (3593 GPs behind) idle=2a4/0/0 softirq=70870/70872 fqs=0 
> [402133.268686]         127-...: (4228 GPs behind) idle=508/0/0 softirq=72137/72139 fqs=0 
> [402133.283438]         (detected by 48, t=2262 jiffies, g=53669, c=53668, q=18005)
> [402133.297060]   CPU[  2]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/2:0]
> [402133.317316]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.342191]   CPU[  3]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/3:0]
> [402133.362453]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.387329]   CPU[  4]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/4:0]
> [402133.407593]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.432470]   CPU[  5]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/5:0]
> [402133.452747]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.477624]   CPU[  6]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/6:0]
> [402133.497885]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.522763]   CPU[  7]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/7:0]
> [402133.543040]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.567920]   CPU[  9]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/9:0]
> [402133.588178]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.613055]   CPU[ 11]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/11:0]
> [402133.633509]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.658386]   CPU[ 12]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/12:0]
> [402133.678823]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.703709]   CPU[ 13]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/13:0]
> [402133.724146]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.749023]   CPU[ 14]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/14:0]
> [402133.769459]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.794336]   CPU[ 15]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/15:0]
> [402133.814773]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.839667]   CPU[ 20]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/20:0]
> [402133.860104]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.884981]   CPU[ 21]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/21:0]
> [402133.905414]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.930298]   CPU[ 22]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/22:0]
> [402133.950747]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.975617]   CPU[ 23]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/23:0]
> [402133.996053]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.020947]   CPU[ 27]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/27:0]
> [402134.041384]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.066269]   CPU[ 28]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/28:0]
> [402134.086705]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.111582]   CPU[ 29]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/29:0]
> [402134.132018]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.156896]   CPU[ 30]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/30:0]
> [402134.177333]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.202229]   CPU[ 31]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/31:0]
> [402134.222666]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.247548]   CPU[ 33]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/33:0]
> [402134.267978]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.292856]   CPU[ 34]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/34:0]
> [402134.313292]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.338186]   CPU[ 35]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/35:0]
> [402134.358625]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.383502]   CPU[ 36]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/36:0]
> [402134.403939]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.428817]   CPU[ 37]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/37:0]
> [402134.449265]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.474142]   CPU[ 38]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/38:0]
> [402134.494572]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.519457]   CPU[ 39]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/39:0]
> [402134.539894]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.564788]   CPU[ 42]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/42:0]
> [402134.585224]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.610095]   CPU[ 43]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/43:0]
> [402134.630548]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.655425]   CPU[ 44]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/44:0]
> [402134.675861]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.700744]   CPU[ 45]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/45:0]
> [402134.721180]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.746057]   CPU[ 46]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/46:0]
> [402134.766494]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.791379]   CPU[ 47]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/47:0]
> [402134.811816]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.836710]   CPU[ 50]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/50:0]
> [402134.857146]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.882023]   CPU[ 51]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/51:0]
> [402134.902460]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.927337]   CPU[ 52]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/52:0]
> [402134.947773]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.972667]   CPU[ 53]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/53:0]
> [402134.993104]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.017987]   CPU[ 54]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/54:0]
> [402135.038424]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.063301]   CPU[ 55]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/55:0]
> [402135.083737]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.108614]   CPU[ 57]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/57:0]
> [402135.129067]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.153945]   CPU[ 59]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/59:0]
> [402135.174381]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.199258]   CPU[ 61]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/61:0]
> [402135.219705]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.244582]   CPU[ 62]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/62:0]
> [402135.265019]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.289896]   CPU[ 63]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/63:0]
> [402135.310332]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.335231]   CPU[ 64]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/64:0]
> [402135.355653]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.380545]   CPU[ 65]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/65:0]
> [402135.400982]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.425859]   CPU[ 66]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/66:0]
> [402135.446295]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.471181]   CPU[ 67]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/67:0]
> [402135.491617]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.516495]   CPU[ 68]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/68:0]
> [402135.536931]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.561825]   CPU[ 69]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/69:0]
> [402135.582262]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.607140]   CPU[ 70]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/70:0]
> [402135.627595]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.652495]   CPU[ 71]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/71:0]
> [402135.672926]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.697815]   CPU[ 72]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/72:0]
> [402135.718231]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.743099]   CPU[ 73]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/73:0]
> [402135.763537]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.788420]   CPU[ 74]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/74:0]
> [402135.808853]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.833748]   CPU[ 75]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/75:0]
> [402135.854184]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.879062]   CPU[ 76]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/76:0]
> [402135.899499]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.924377]   CPU[ 77]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/77:0]
> [402135.944813]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.969707]   CPU[ 78]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/78:0]
> [402135.990144]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.015022]   CPU[ 79]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/79:0]
> [402136.035460]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.060338]   CPU[ 80]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/80:0]
> [402136.080783]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.105667]   CPU[ 81]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/81:0]
> [402136.126100]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.150977]   CPU[ 82]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/82:0]
> [402136.171413]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.196308]   CPU[ 83]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/83:0]
> [402136.216745]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.241623]   CPU[ 84]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/84:0]
> [402136.262059]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.286937]   CPU[ 85]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/85:0]
> [402136.307373]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.332267]   CPU[ 86]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/86:0]
> [402136.352704]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.377586]   CPU[ 87]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/87:0]
> [402136.398023]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.422900]   CPU[ 88]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/88:0]
> [402136.443337]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.468213]   CPU[ 89]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/89:0]
> [402136.488667]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.513544]   CPU[ 90]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/90:0]
> [402136.533981]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.558859]   CPU[ 91]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/91:0]
> [402136.579304]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.604182]   CPU[ 92]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/92:0]
> [402136.624619]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.649496]   CPU[ 93]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/93:0]
> [402136.669933]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.694827]   CPU[ 94]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/94:0]
> [402136.715264]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.740141]   CPU[ 95]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/95:0]
> [402136.760578]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.785459]   CPU[ 96]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/96:0]
> [402136.805895]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.830787]   CPU[ 97]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/97:0]
> [402136.851212]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.876094]   CPU[ 98]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/98:0]
> [402136.896547]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.921424]   CPU[ 99]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/99:0]
> [402136.941863]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.966741]   CPU[100]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/100:0]
> [402136.987359]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.012237]   CPU[101]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/101:0]
> [402137.032846]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.057723]   CPU[102]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/102:0]
> [402137.078336]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.103230]   CPU[103]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/103:0]
> [402137.123826]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.148728]   CPU[104]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/104:0]
> [402137.169327]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.194204]   CPU[105]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/105:0]
> [402137.214817]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.239703]   CPU[106]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/106:0]
> [402137.260316]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.285194]   CPU[107]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/107:0]
> [402137.305807]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.330685]   CPU[108]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/108:0]
> [402137.351297]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.376192]   CPU[109]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/109:0]
> [402137.396788]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.421682]   CPU[110]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/110:0]
> [402137.442297]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.467178]   CPU[111]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/111:0]
> [402137.487775]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.512669]   CPU[112]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/112:0]
> [402137.533279]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.558158]   CPU[113]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/113:0]
> [402137.578773]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.603650]   CPU[114]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/114:0]
> [402137.624265]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.649142]   CPU[115]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/115:0]
> [402137.669755]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.694643]   CPU[116]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/116:0]
> [402137.715256]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.740133]   CPU[117]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/117:0]
> [402137.760747]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.785624]   CPU[118]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/118:0]
> [402137.806236]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.831116]   CPU[119]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/119:0]
> [402137.851730]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.876610]   CPU[120]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/120:0]
> [402137.897223]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.922100]   CPU[121]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/121:0]
> [402137.942713]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.967596]   CPU[122]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/122:0]
> [402137.988209]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402138.013099]   CPU[123]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/123:0]
> [402138.033714]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.058584]   CPU[124]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/124:0]
> [402138.079197]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.104074]   CPU[125]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/125:0]
> [402138.124687]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.149583]   CPU[126]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/126:0]
> [402138.170183]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.195078]   CPU[127]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/127:0]
> [402138.215685]              TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
> [402138.259779] rcu_sched       S    0     9      2 0x06000000
> [402138.270887] Call Trace:
> [402138.275922]  [00000000009af81c] schedule+0x1c/0xa0
> [402138.285646]  [00000000009b2ac0] schedule_timeout+0x1a0/0x340
> [402138.297110]  [00000000004c3dcc] rcu_gp_kthread+0x68c/0xa60
> [402138.308222]  [000000000048dd14] kthread+0xd4/0x120
> [402138.317959]  [0000000000406064] ret_from_fork+0x1c/0x2c
> [402138.328537]  [0000000000000000]           (null)
> 
>