Re: TREE_SRCU slows hotplug by factor ~16
From: Paul E. McKenney
Date: Mon Apr 24 2017 - 12:24:56 EST
On Mon, Apr 24, 2017 at 09:35:03AM +0200, Mike Galbraith wrote:
> On Sun, 2017-04-23 at 23:22 -0700, Paul E. McKenney wrote:
>
> > Could you please collect an ftrace (or whatever) showing the timestamp
> > sequence of calls to synchronize_srcu(), synchronize_srcu_expedited(),
> > and call_srcu() during the execution of the stress script? If it is easy
> > to do, also the timestamp sequence of returns from synchronize_srcu()
> > and synchronize_srcu_expedited()?
>
> The first two minutes below config bits, trace_printk([enter/exit]) in
> each function. If you (unlikely, but..) want the whole trace, holler.
>
> # RCU Subsystem
> CONFIG_TREE_RCU=y
> CONFIG_RCU_EXPERT=y
> CONFIG_SRCU=y
> # CONFIG_CLASSIC_SRCU is not set
> CONFIG_TREE_SRCU=y
> # CONFIG_TASKS_RCU is not set
> CONFIG_RCU_STALL_COMMON=y
> CONFIG_RCU_FANOUT=64
> CONFIG_RCU_FANOUT_LEAF=16
> # CONFIG_RCU_FAST_NO_HZ is not set
> # CONFIG_TREE_RCU_TRACE is not set
> CONFIG_RCU_KTHREAD_PRIO=0
> CONFIG_RCU_NOCB_CPU=y
> CONFIG_RCU_NOCB_CPU_NONE=y
> # CONFIG_RCU_NOCB_CPU_ZERO is not set
> # CONFIG_RCU_NOCB_CPU_ALL is not set
> # RCU Debugging
> # CONFIG_PROVE_RCU is not set
> # CONFIG_SPARSE_RCU_POINTER is not set
> # CONFIG_RCU_PERF_TEST is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> CONFIG_RCU_CPU_STALL_TIMEOUT=60
> # CONFIG_RCU_TRACE is not set
> # CONFIG_RCU_EQS_DEBUG is not set
Thank you for the data, extremely helpful! I now know that I must
immediately fix this the hard way rather than trying several simpler
possibilities that, according to your trace data, would be complete
wastes of time.
My initial calculation was actually optimistic. There was one
exit-to-enter interval at about 80 milliseconds, but the second
longest is 342 microseconds, and the vast majority are under 200
microseconds.
Back to the drawing board!
This will take some time, but I should have a patch for you in a few days,
hopefully sooner.
Thanx, Paul
> # tracer: nop
> #
> # entries-in-buffer/entries-written: 229332/229332 #P:8
> #
> # _-----=> irqs-off
> # / _----=> need-resched
> # | / _---=> hardirq/softirq
> # || / _--=> preempt-depth
> # ||| / delay
> # TASK-PID CPU# |||| TIMESTAMP FUNCTION
> # | | | |||| | |
> stress-cpu-hotp-2443 [004] .... 144.123608: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.123610: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.123612: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.139528: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.139646: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.139646: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.139648: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.155813: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.155981: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.155982: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.155983: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.172078: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.172234: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.172235: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.172237: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.188319: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.188465: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.188466: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.188468: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.202528: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.202650: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.202651: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.202652: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.219081: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.219238: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.219238: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.219240: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.235319: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.235567: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.235568: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.235571: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.251548: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.251706: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.251706: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.251708: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.267865: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.268132: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.268133: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.268136: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.284078: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.284257: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.284258: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.284260: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.300314: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.300462: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.300463: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.300464: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.314533: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.314657: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.314658: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.314659: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.331056: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.331227: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.331227: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.331229: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.347334: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.347517: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.347518: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.347519: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.363567: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.363731: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.363732: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.363733: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.379804: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.379987: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.379988: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.379989: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.396077: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.396259: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.396260: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.396261: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.412385: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.412653: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.412654: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.412657: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.426647: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.426888: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.426889: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.426891: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.443134: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.443325: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.443325: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.443327: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.459327: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.459521: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.459522: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.459524: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.475553: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.475672: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.475673: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.475674: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.491798: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.491919: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.491920: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.491921: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.508035: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.508160: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.508161: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.508162: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.524301: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.524419: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.524419: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.524421: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.538545: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.538665: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.538666: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.538667: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.555051: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.555174: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.555174: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.555175: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.571290: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.571412: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.571412: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.571413: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.587554: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.587672: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.587673: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.587674: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.603843: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.604008: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.604008: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.604010: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.620087: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.620317: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.620318: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.620321: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.636342: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.636520: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.636521: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.636523: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.650579: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.650737: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.650738: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.650739: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.667084: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.667247: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.667247: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.667249: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.683340: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.683513: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.683514: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.683516: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.699556: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.699675: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.699675: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.699676: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.715841: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.716013: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.716014: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.716015: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.732096: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.732269: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.732270: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.732271: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.748393: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.748641: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.748642: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.748645: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.762639: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.762854: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.762855: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.762857: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.779100: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.779278: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.779279: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.779280: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.795348: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.795525: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.795526: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.795527: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.811583: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.811749: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.811750: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.811751: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.827831: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.827997: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.827998: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.827999: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.844104: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.844281: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.844281: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.844283: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.860339: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.860484: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.860485: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.860487: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.874584: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.874737: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.874737: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.874739: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.891117: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 144.891459: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.891460: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 144.891463: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.907376: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 144.907628: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.907629: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 144.907631: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.923590: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 144.923756: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.923756: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 144.923758: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.939837: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 144.940007: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.940008: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 144.940009: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.956110: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 144.956285: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.956286: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 144.956287: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.972344: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 144.972492: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.972493: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 144.972495: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.986566: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 144.986689: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.986689: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 144.986690: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.003114: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.003292: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.003293: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.003294: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.019350: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.019528: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.019529: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.019531: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.035659: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.035939: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.035940: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.035943: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.051891: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.052106: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.052107: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.052109: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.068125: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.068279: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.068279: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.068280: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.084332: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.084472: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.084473: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.084474: call_srcu: exit
> stress-cpu-hotp-2443 [002] .... 145.098605: synchronize_srcu: exit
> stress-cpu-hotp-2443 [002] .... 145.098743: synchronize_srcu: enter
> stress-cpu-hotp-2443 [002] .... 145.098744: call_srcu: enter
> stress-cpu-hotp-2443 [002] .... 145.098745: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.114590: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.114728: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.114728: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.114729: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.130591: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.130728: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.130728: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.130730: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.147857: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.228078: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.228079: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.228080: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.244025: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.244106: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.244106: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.244107: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.260286: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.260361: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.260361: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.260362: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.274522: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.274594: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.274594: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.274595: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.291289: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.291358: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.291358: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.291359: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.307529: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.307599: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.307599: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.307600: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.323901: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.324063: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.324063: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.324065: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.340122: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.340288: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.340289: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.340291: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.356378: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.356519: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.356520: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.356521: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.370650: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.370797: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.370798: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.370799: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.387385: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.387547: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.387548: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.387549: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.403636: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.403799: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.403800: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.403802: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.419885: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.420055: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.420056: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.420057: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.436154: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.436372: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.436372: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.436374: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.452413: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.452550: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.452551: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.452552: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.466606: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.466721: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.466721: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.466722: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.483395: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.483555: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.483555: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.483557: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.499628: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.499761: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .N.. 145.499774: call_srcu: enter
> stress-cpu-hotp-2443 [005] .N.. 145.499775: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.515879: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.516009: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.516009: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.516011: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.532164: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.532488: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.532489: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.532492: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.548447: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.548659: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.548660: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.548663: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.562706: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.562897: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.562898: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.562900: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.579403: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.579562: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.579563: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.579564: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.595636: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.595766: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.595766: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.595768: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.611902: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.612059: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.612060: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.612061: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.628136: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.628266: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.628267: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.628268: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.644392: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.644518: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.644519: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.644521: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.658609: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.658698: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.658699: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.658700: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.675410: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.675570: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.675571: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.675572: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.691706: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.691950: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.691951: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.691953: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.707926: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.708096: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.708097: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.708099: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.724154: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.724333: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.724333: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.724335: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.740412: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.740559: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.740559: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.740561: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.754685: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.754835: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.754836: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.754838: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.771415: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.771591: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.771592: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.771594: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.787635: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.787747: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.787748: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.787749: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.803943: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.804168: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.804170: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.804172: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.820168: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.820320: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.820321: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.820322: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.836412: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.836543: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.836544: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.836545: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.850638: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.850757: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.850757: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.850758: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.867433: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.867597: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.867598: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.867599: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.883680: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.883843: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.883843: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.883845: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.899908: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.900073: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.900073: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.900075: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.916213: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 145.916405: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.916406: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 145.916408: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.932453: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 145.932653: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.932653: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 145.932656: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.946674: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 145.946818: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.946819: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 145.946820: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.963402: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 145.963539: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.963540: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 145.963542: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.979671: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 145.979790: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.979790: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 145.979792: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.995927: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 145.996046: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.996046: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 145.996048: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 146.012167: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 146.012282: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 146.012283: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 146.012284: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 146.028405: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 146.028522: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 146.028522: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 146.028523: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 146.042653: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 146.042774: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 146.042774: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 146.042775: call_srcu: exit
> stress-cpu-hotp-2443 [004] .... 146.059407: synchronize_srcu: exit
> stress-cpu-hotp-2443 [004] .... 146.059521: synchronize_srcu: enter
> stress-cpu-hotp-2443 [004] .... 146.059522: call_srcu: enter
> stress-cpu-hotp-2443 [004] .... 146.059523: call_srcu: exit
> stress-cpu-hotp-2443 [005] .... 146.075658: synchronize_srcu: exit
> stress-cpu-hotp-2443 [005] .... 146.075772: synchronize_srcu: enter
> stress-cpu-hotp-2443 [005] .... 146.075773: call_srcu: enter
> stress-cpu-hotp-2443 [005] .... 146.075774: call_srcu: exit
> stress-cpu-hotp-2443 [006] .... 146.091906: synchronize_srcu: exit
> stress-cpu-hotp-2443 [006] .... 146.092021: synchronize_srcu: enter
> stress-cpu-hotp-2443 [006] .... 146.092022: call_srcu: enter
> stress-cpu-hotp-2443 [006] .... 146.092023: call_srcu: exit
> stress-cpu-hotp-2443 [007] .... 146.108187: synchronize_srcu: exit
> stress-cpu-hotp-2443 [007] .... 146.108336: synchronize_srcu: enter
> stress-cpu-hotp-2443 [007] .... 146.108337: call_srcu: enter
> stress-cpu-hotp-2443 [007] .... 146.108338: call_srcu: exit
> stress-cpu-hotp-2443 [000] .... 146.124439: synchronize_srcu: exit
> stress-cpu-hotp-2443 [000] .... 146.124572: synchronize_srcu: enter
> stress-cpu-hotp-2443 [000] .... 146.124572: call_srcu: enter
> stress-cpu-hotp-2443 [000] .... 146.124574: call_srcu: exit
> stress-cpu-hotp-2443 [003] .... 146.138656: synchronize_srcu: exit
> stress-cpu-hotp-2443 [003] .... 146.138773: synchronize_srcu: enter
> stress-cpu-hotp-2443 [003] .... 146.138773: call_srcu: enter
> stress-cpu-hotp-2443 [003] .... 146.138774: call_srcu: exit
>
>