RE: [PATCH] rcu: Add exp QS check in rcu_exp_handler() for no-preemptible expedited RCU
From: Zhang, Qiang1
Date: Thu Jun 23 2022 - 00:47:40 EST
On Wed, Jun 22, 2022 at 05:34:09PM -0700, Paul E. McKenney wrote:
> On Wed, Jun 22, 2022 at 11:34:15PM +0000, Zhang, Qiang1 wrote:
> >
> > On Wed, Jun 22, 2022 at 06:35:49PM +0800, Zqiang wrote:
> > > In CONFIG_PREEMPT=n and CONFIG_PREEMPT_COUNT=y kernel, after a exp
> > > grace period begins, if detected current CPU enters idle in
> > > rcu_exp_handler() IPI handler, will immediately report the exp QS of the
> > > current cpu, at this time, maybe not being in an RCU read-side critical
> > > section, but need wait until rcu-softirq or sched-clock irq or sched-switch
> > > occurs on current CPU to check and report exp QS.
> > >
> >
> > >I think the idea is OK, however, this "optimization" is based on the
> > >implementation detail that rcu_read_lock() counts preempt_count when
> > >CONFIG_PREEMPT_COUNT=y, right? It's a little bit dangerous because the
> > >preempt_count when CONFIG_PREEMPT_COUNT=y and CONFIG_PREEMPT=n is mostly
> > >for debugging purposes IIUC, and in other words, _it could be gone_.
> > >
> >
> > Yes, for CONFIG_PREEMPT_COUNT=y and CONFIG_PREEMPT=n kernel
> > The rcu_read_lock/unlock are replaced by preempt_disbale/enable, and the
> > preempt-count is exists, so can report exp QS when not being an RCU
> > read-side critical(preempt_count & (PREEMPT_MASK | SOFTIRQ_MASK )return zero).
> > in IPI handler.
> >
> > For CONFIG_PREEMPT_COUNT=n and CONFIG_PREEMPT=n kernel,
> > The rcu_read_lock/unlock is just barrier().
> >
> >
> > So I add IS_ENABLED(CONFIG_PREEMPT_COUNT) check in code.
> >
> > Of course, for CONFIG_PREEMPT_COUNT=n kernel, in RCU softirq, the
> > preempt-count is also checked
> >
> > /* Report any deferred quiescent states if preemption enabled. */
> > if (IS_ENABLED(CONFIG_PREEMPT_COUNT) && (!(preempt_count() & PREEMPT_MASK))) {
> > rcu_preempt_deferred_qs(current);
> >
> > but the RCU softirq may not be triggered in time and reported exp QS, for
> > example a kernel loop exist on NO_HZ_FULL CPU
> >
> > this change, It is to capture the exp QS state earlier and report it.
> >
> >
> > >Also I'm not aware of any but there could be someone assuming that RCU
> > >read-side critical sections can be formed without
> > >rcu_read_{lock,unlock}() in CONFIG_PREEMPT=n kernel. For example, there
> > >might be "creative" code like the following:
> > >
> > > void do_something_only_in_nonpreempt(void)
> > > {
> > > int *p;
> > >
> > > // This function only gets called in PREEMPT=n kernel,
> > > // which means everywhere is a RCU read-side critical
> > > // section, let's save some lines of code.
> > >
> > rcu_read_lock();
> > > p = rcu_dereference_check(gp, !IS_ENABLED(PREEMPT));
> > > ... // of course no schedule() here.
> > > <access p>
> > rcu_read_unlock();
> > > }
> > >
> >
> > Usually access to pointers of type rcu needs to be protected.
Yes, _ususally_ they are, but what about the special cases? Because
in PREEMPT=n kernel, almost everywhere is a RCU read-side critical
section, some one might have been "creative" enough to omit these
rcu_read_lock() and rcu_read_unlock().
>
> Indeed, lockdep would normally complain about this sort of thing.
> But in kernels built with (say) CONFIG_PREEMPT_NONE=y but without
> CONFIG_PREEMPT_COUNT=N, can lockdep really tell the difference?
>
>Actually with the rcu_dereference_check() above, lockdep cannot detect
>even CONFIG_PREEMPT_COUNT=y, that rcu_dereference_check() basically says
>"I know I'm in a read-side critical section if it's a non-preempt
>kernel, so don't bother to check". ;-)
> > Any thoughts?
>
> It would be good to have some performance data on this change to expedited
> grace periods. It is adding code, so it needs some real motivation.
I used rcuscale to test it, and count the average time of writer-duration
no applied patch(org.txt)
writer avg time 29690.39 29670.78 29770.65 29423.25
applied patch(new.txt)
writer avg time 28989.99 29003.54 29281.39 28986.58
or Is there a better way to test?
Thanks
Zqiang
>Agreed.
>
>Regards,
>Boqun
> So, does this change make a system-level difference in (say) expedited
> RCU grace-period latency, and if so, under what conditions?
>
> Thanx, Paul
>
> > >Again, I'm not aware of any existing code that does this but we need to
> > >be sure.
> > >
> > >Regards,
> > >Boqun
> > >
> > > This commit add a exp QS check in rcu_exp_handler(), when not being
> > > in an RCU read-side critical section, report exp QS earlier.
> > >
> > > Signed-off-by: Zqiang <qiang1.zhang@xxxxxxxxx>
> > > ---
> > > kernel/rcu/tree_exp.h | 5 ++++-
> > > 1 file changed, 4 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> > > index be667583a554..34f08267410f 100644
> > > --- a/kernel/rcu/tree_exp.h
> > > +++ b/kernel/rcu/tree_exp.h
> > > @@ -828,11 +828,14 @@ static void rcu_exp_handler(void *unused)
> > > {
> > > struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
> > > struct rcu_node *rnp = rdp->mynode;
> > > + bool preempt_bh_disabled =
> > > + !!(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK));
> > >
> > > if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
> > > __this_cpu_read(rcu_data.cpu_no_qs.b.exp))
> > > return;
> > > - if (rcu_is_cpu_rrupt_from_idle()) {
> > > + if (rcu_is_cpu_rrupt_from_idle() ||
> > > + (IS_ENABLED(CONFIG_PREEMPT_COUNT) && !preempt_bh_disabled)) {
> > > rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
> > > return;
> > > }
> > > --
> > > 2.25.1
> > >
[ 28.907109] rcu-scale: rcu_scale_writer 3 has 100 measurements
[ 28.907133] rcu-scale: rcu_scale_writer 0 has 100 measurements
[ 28.907135] rcu-scale: rcu_scale_writer 1 has 100 measurements
[ 28.907138] rcu-scale: rcu_scale_writer 2 has 100 measurements
[ 28.917241] Dumping ftrace buffer:
[ 28.917256] (ftrace buffer empty)
[ 28.917261] rcu-scale: Test complete
[ 49.336302] rcu-scale:!!! All grace periods expedited, no normal ones to measure!
[ 49.336329] rcu-torture: Stopping rcu_scale_reader task
[ 49.336340] rcu-torture: rcu_scale_reader is stopping
[ 49.336351] rcu-torture: Stopping rcu_scale_reader task
[ 49.336361] rcu-torture: rcu_scale_reader is stopping
[ 49.336370] rcu-torture: Stopping rcu_scale_reader task
[ 49.336405] rcu-torture: rcu_scale_reader is stopping
[ 49.336426] rcu-torture: Stopping rcu_scale_reader task
[ 49.336432] rcu-torture: rcu_scale_reader is stopping
[ 49.336443] rcu-torture: Stopping rcu_scale_writer task
[ 49.336450] rcu-torture: rcu_scale_writer is stopping
[ 49.336458] rcu-scale: writer 0 gps: 102
[ 49.336460] rcu-torture: Stopping rcu_scale_writer task
[ 49.336465] rcu-torture: rcu_scale_writer is stopping
[ 49.336517] rcu-scale: writer 1 gps: 102
[ 49.336519] rcu-torture: Stopping rcu_scale_writer task
[ 49.336524] rcu-torture: rcu_scale_writer is stopping
[ 49.336532] rcu-scale: writer 2 gps: 101
[ 49.336533] rcu-torture: Stopping rcu_scale_writer task
[ 49.336539] rcu-torture: rcu_scale_writer is stopping
[ 49.336546] rcu-scale: writer 3 gps: 102
[ 49.336548] rcu-scale: start: 28841772292 end: 28844718545 duration: 2946253 gps: 407 batches: 2
[ 49.336550] rcu-scale: 0 writer-duration: 0 25552
[ 49.339264] rcu-scale: 0 writer-duration: 1 42012
[ 49.339274] rcu-scale: 0 writer-duration: 2 24377
[ 49.339280] rcu-scale: 0 writer-duration: 3 34458
[ 49.339299] rcu-scale: 0 writer-duration: 4 27007
[ 49.339305] rcu-scale: 0 writer-duration: 5 36494
[ 49.339310] rcu-scale: 0 writer-duration: 6 24278
[ 49.339315] rcu-scale: 0 writer-duration: 7 29383
[ 49.339321] rcu-scale: 0 writer-duration: 8 26494
[ 49.339326] rcu-scale: 0 writer-duration: 9 35430
[ 49.339331] rcu-scale: 0 writer-duration: 10 23986
[ 49.339337] rcu-scale: 0 writer-duration: 11 28663
[ 49.339342] rcu-scale: 0 writer-duration: 12 26329
[ 49.339347] rcu-scale: 0 writer-duration: 13 35645
[ 49.339353] rcu-scale: 0 writer-duration: 14 23761
[ 49.339358] rcu-scale: 0 writer-duration: 15 28889
[ 49.339364] rcu-scale: 0 writer-duration: 16 26519
[ 49.339369] rcu-scale: 0 writer-duration: 17 35863
[ 49.339374] rcu-scale: 0 writer-duration: 18 23984
[ 49.339380] rcu-scale: 0 writer-duration: 19 28618
[ 49.339385] rcu-scale: 0 writer-duration: 20 26547
[ 49.339390] rcu-scale: 0 writer-duration: 21 35515
[ 49.339395] rcu-scale: 0 writer-duration: 22 23913
[ 49.339400] rcu-scale: 0 writer-duration: 23 28510
[ 49.339405] rcu-scale: 0 writer-duration: 24 26746
[ 49.339410] rcu-scale: 0 writer-duration: 25 35670
[ 49.339416] rcu-scale: 0 writer-duration: 26 23963
[ 49.339421] rcu-scale: 0 writer-duration: 27 28561
[ 49.339427] rcu-scale: 0 writer-duration: 28 26380
[ 49.339432] rcu-scale: 0 writer-duration: 29 36039
[ 49.339438] rcu-scale: 0 writer-duration: 30 24215
[ 49.339443] rcu-scale: 0 writer-duration: 31 28891
[ 49.339449] rcu-scale: 0 writer-duration: 32 26578
[ 49.339454] rcu-scale: 0 writer-duration: 33 40498
[ 49.339459] rcu-scale: 0 writer-duration: 34 24245
[ 49.339464] rcu-scale: 0 writer-duration: 35 29465
[ 49.339469] rcu-scale: 0 writer-duration: 36 26946
[ 49.339475] rcu-scale: 0 writer-duration: 37 35955
[ 49.339480] rcu-scale: 0 writer-duration: 38 24059
[ 49.339486] rcu-scale: 0 writer-duration: 39 28795
[ 49.339491] rcu-scale: 0 writer-duration: 40 26536
[ 49.339496] rcu-scale: 0 writer-duration: 41 35879
[ 49.339501] rcu-scale: 0 writer-duration: 42 24181
[ 49.339506] rcu-scale: 0 writer-duration: 43 28639
[ 49.339512] rcu-scale: 0 writer-duration: 44 26546
[ 49.339517] rcu-scale: 0 writer-duration: 45 35884
[ 49.339522] rcu-scale: 0 writer-duration: 46 23986
[ 49.339527] rcu-scale: 0 writer-duration: 47 28668
[ 49.339532] rcu-scale: 0 writer-duration: 48 26454
[ 49.339537] rcu-scale: 0 writer-duration: 49 35772
[ 49.339542] rcu-scale: 0 writer-duration: 50 23942
[ 49.339547] rcu-scale: 0 writer-duration: 51 28803
[ 49.339553] rcu-scale: 0 writer-duration: 52 26545
[ 49.339558] rcu-scale: 0 writer-duration: 53 35615
[ 49.339563] rcu-scale: 0 writer-duration: 54 23569
[ 49.339568] rcu-scale: 0 writer-duration: 55 28427
[ 49.339573] rcu-scale: 0 writer-duration: 56 26489
[ 49.339578] rcu-scale: 0 writer-duration: 57 35516
[ 49.339584] rcu-scale: 0 writer-duration: 58 23722
[ 49.339589] rcu-scale: 0 writer-duration: 59 28435
[ 49.339594] rcu-scale: 0 writer-duration: 60 26401
[ 49.339599] rcu-scale: 0 writer-duration: 61 35488
[ 49.339604] rcu-scale: 0 writer-duration: 62 23855
[ 49.339610] rcu-scale: 0 writer-duration: 63 28583
[ 49.339615] rcu-scale: 0 writer-duration: 64 26629
[ 49.339620] rcu-scale: 0 writer-duration: 65 36168
[ 49.339625] rcu-scale: 0 writer-duration: 66 24191
[ 49.339630] rcu-scale: 0 writer-duration: 67 28802
[ 49.339635] rcu-scale: 0 writer-duration: 68 32436
[ 49.339640] rcu-scale: 0 writer-duration: 69 38444
[ 49.339646] rcu-scale: 0 writer-duration: 70 24417
[ 49.339651] rcu-scale: 0 writer-duration: 71 29128
[ 49.339656] rcu-scale: 0 writer-duration: 72 26480
[ 49.339661] rcu-scale: 0 writer-duration: 73 35881
[ 49.339666] rcu-scale: 0 writer-duration: 74 24123
[ 49.339671] rcu-scale: 0 writer-duration: 75 28735
[ 49.339676] rcu-scale: 0 writer-duration: 76 26359
[ 49.339681] rcu-scale: 0 writer-duration: 77 35715
[ 49.339687] rcu-scale: 0 writer-duration: 78 24092
[ 49.339692] rcu-scale: 0 writer-duration: 79 28716
[ 49.339698] rcu-scale: 0 writer-duration: 80 26448
[ 49.339703] rcu-scale: 0 writer-duration: 81 35935
[ 49.339708] rcu-scale: 0 writer-duration: 82 23966
[ 49.339713] rcu-scale: 0 writer-duration: 83 28552
[ 49.339718] rcu-scale: 0 writer-duration: 84 26487
[ 49.339723] rcu-scale: 0 writer-duration: 85 35490
[ 49.339729] rcu-scale: 0 writer-duration: 86 23976
[ 49.339734] rcu-scale: 0 writer-duration: 87 28679
[ 49.339740] rcu-scale: 0 writer-duration: 88 26525
[ 49.339745] rcu-scale: 0 writer-duration: 89 35764
[ 49.339750] rcu-scale: 0 writer-duration: 90 24050
[ 49.339756] rcu-scale: 0 writer-duration: 91 28777
[ 49.339761] rcu-scale: 0 writer-duration: 92 26412
[ 49.339767] rcu-scale: 0 writer-duration: 93 35707
[ 49.339772] rcu-scale: 0 writer-duration: 94 23913
[ 49.339777] rcu-scale: 0 writer-duration: 95 28450
[ 49.339782] rcu-scale: 0 writer-duration: 96 26665
[ 49.339788] rcu-scale: 0 writer-duration: 97 35636
[ 49.339793] rcu-scale: 0 writer-duration: 98 23953
[ 49.339798] rcu-scale: 0 writer-duration: 99 28466
[ 49.339804] rcu-scale: 0 writer-duration: 100 26654
[ 49.341227] rcu-scale: 0 writer-duration: 101 225635
[ 49.341246] rcu-scale: 1 writer-duration: 0 27277
[ 49.343227] rcu-scale: 1 writer-duration: 1 34458
[ 49.343236] rcu-scale: 1 writer-duration: 2 27340
[ 49.343242] rcu-scale: 1 writer-duration: 3 41758
[ 49.343248] rcu-scale: 1 writer-duration: 4 24161
[ 49.343254] rcu-scale: 1 writer-duration: 5 29250
[ 49.343260] rcu-scale: 1 writer-duration: 6 26996
[ 49.343266] rcu-scale: 1 writer-duration: 7 36482
[ 49.343272] rcu-scale: 1 writer-duration: 8 23782
[ 49.343277] rcu-scale: 1 writer-duration: 9 28433
[ 49.343283] rcu-scale: 1 writer-duration: 10 26577
[ 49.343289] rcu-scale: 1 writer-duration: 11 35616
[ 49.343295] rcu-scale: 1 writer-duration: 12 23821
[ 49.343301] rcu-scale: 1 writer-duration: 13 28678
[ 49.343307] rcu-scale: 1 writer-duration: 14 26517
[ 49.343313] rcu-scale: 1 writer-duration: 15 35548
[ 49.343319] rcu-scale: 1 writer-duration: 16 24196
[ 49.343325] rcu-scale: 1 writer-duration: 17 28626
[ 49.343332] rcu-scale: 1 writer-duration: 18 26772
[ 49.343338] rcu-scale: 1 writer-duration: 19 35539
[ 49.343344] rcu-scale: 1 writer-duration: 20 23897
[ 49.343350] rcu-scale: 1 writer-duration: 21 28635
[ 49.343356] rcu-scale: 1 writer-duration: 22 26453
[ 49.343362] rcu-scale: 1 writer-duration: 23 35416
[ 49.343368] rcu-scale: 1 writer-duration: 24 24096
[ 49.343373] rcu-scale: 1 writer-duration: 25 28733
[ 49.343379] rcu-scale: 1 writer-duration: 26 26538
[ 49.343385] rcu-scale: 1 writer-duration: 27 35590
[ 49.343390] rcu-scale: 1 writer-duration: 28 23739
[ 49.343396] rcu-scale: 1 writer-duration: 29 29450
[ 49.343402] rcu-scale: 1 writer-duration: 30 26452
[ 49.343408] rcu-scale: 1 writer-duration: 31 35807
[ 49.343413] rcu-scale: 1 writer-duration: 32 24145
[ 49.343419] rcu-scale: 1 writer-duration: 33 33452
[ 49.343425] rcu-scale: 1 writer-duration: 34 26767
[ 49.343431] rcu-scale: 1 writer-duration: 35 36677
[ 49.343438] rcu-scale: 1 writer-duration: 36 24102
[ 49.343444] rcu-scale: 1 writer-duration: 37 28937
[ 49.343449] rcu-scale: 1 writer-duration: 38 26622
[ 49.343454] rcu-scale: 1 writer-duration: 39 35903
[ 49.343460] rcu-scale: 1 writer-duration: 40 23956
[ 49.343465] rcu-scale: 1 writer-duration: 41 28684
[ 49.343470] rcu-scale: 1 writer-duration: 42 26901
[ 49.343475] rcu-scale: 1 writer-duration: 43 35717
[ 49.343480] rcu-scale: 1 writer-duration: 44 23957
[ 49.343485] rcu-scale: 1 writer-duration: 45 28780
[ 49.343490] rcu-scale: 1 writer-duration: 46 26747
[ 49.343495] rcu-scale: 1 writer-duration: 47 35538
[ 49.343501] rcu-scale: 1 writer-duration: 48 23963
[ 49.343506] rcu-scale: 1 writer-duration: 49 28727
[ 49.343511] rcu-scale: 1 writer-duration: 50 26519
[ 49.343516] rcu-scale: 1 writer-duration: 51 35733
[ 49.343521] rcu-scale: 1 writer-duration: 52 24098
[ 49.343527] rcu-scale: 1 writer-duration: 53 28557
[ 49.343532] rcu-scale: 1 writer-duration: 54 26208
[ 49.343537] rcu-scale: 1 writer-duration: 55 35315
[ 49.343543] rcu-scale: 1 writer-duration: 56 23932
[ 49.343548] rcu-scale: 1 writer-duration: 57 28507
[ 49.343553] rcu-scale: 1 writer-duration: 58 26443
[ 49.343559] rcu-scale: 1 writer-duration: 59 35306
[ 49.343564] rcu-scale: 1 writer-duration: 60 23868
[ 49.343570] rcu-scale: 1 writer-duration: 61 28606
[ 49.343575] rcu-scale: 1 writer-duration: 62 26308
[ 49.343580] rcu-scale: 1 writer-duration: 63 35693
[ 49.343586] rcu-scale: 1 writer-duration: 64 23934
[ 49.343591] rcu-scale: 1 writer-duration: 65 29456
[ 49.343596] rcu-scale: 1 writer-duration: 66 26600
[ 49.343602] rcu-scale: 1 writer-duration: 67 35682
[ 49.343607] rcu-scale: 1 writer-duration: 68 29849
[ 49.343612] rcu-scale: 1 writer-duration: 69 31555
[ 49.343618] rcu-scale: 1 writer-duration: 70 26942
[ 49.343623] rcu-scale: 1 writer-duration: 71 36037
[ 49.343629] rcu-scale: 1 writer-duration: 72 23965
[ 49.343634] rcu-scale: 1 writer-duration: 73 28811
[ 49.343640] rcu-scale: 1 writer-duration: 74 26734
[ 49.343645] rcu-scale: 1 writer-duration: 75 35712
[ 49.343650] rcu-scale: 1 writer-duration: 76 23845
[ 49.343656] rcu-scale: 1 writer-duration: 77 28619
[ 49.343661] rcu-scale: 1 writer-duration: 78 26637
[ 49.343666] rcu-scale: 1 writer-duration: 79 35773
[ 49.343672] rcu-scale: 1 writer-duration: 80 23910
[ 49.343677] rcu-scale: 1 writer-duration: 81 28873
[ 49.343682] rcu-scale: 1 writer-duration: 82 26581
[ 49.343688] rcu-scale: 1 writer-duration: 83 35517
[ 49.343694] rcu-scale: 1 writer-duration: 84 23928
[ 49.343699] rcu-scale: 1 writer-duration: 85 28565
[ 49.343704] rcu-scale: 1 writer-duration: 86 26464
[ 49.343710] rcu-scale: 1 writer-duration: 87 35715
[ 49.343715] rcu-scale: 1 writer-duration: 88 23968
[ 49.343721] rcu-scale: 1 writer-duration: 89 28784
[ 49.343726] rcu-scale: 1 writer-duration: 90 26576
[ 49.343731] rcu-scale: 1 writer-duration: 91 35800
[ 49.343737] rcu-scale: 1 writer-duration: 92 23918
[ 49.343742] rcu-scale: 1 writer-duration: 93 28487
[ 49.343747] rcu-scale: 1 writer-duration: 94 26491
[ 49.343753] rcu-scale: 1 writer-duration: 95 35565
[ 49.343758] rcu-scale: 1 writer-duration: 96 24092
[ 49.343763] rcu-scale: 1 writer-duration: 97 28469
[ 49.343768] rcu-scale: 1 writer-duration: 98 26865
[ 49.343774] rcu-scale: 1 writer-duration: 99 35343
[ 49.343779] rcu-scale: 1 writer-duration: 100 23995
[ 49.345221] rcu-scale: 1 writer-duration: 101 60524
[ 49.345235] rcu-scale: 2 writer-duration: 0 56581
[ 49.347314] rcu-scale: 2 writer-duration: 1 32281
[ 49.347329] rcu-scale: 2 writer-duration: 2 29516
[ 49.347336] rcu-scale: 2 writer-duration: 3 29979
[ 49.347342] rcu-scale: 2 writer-duration: 4 36232
[ 49.347348] rcu-scale: 2 writer-duration: 5 26878
[ 49.347354] rcu-scale: 2 writer-duration: 6 29099
[ 49.347359] rcu-scale: 2 writer-duration: 7 24890
[ 49.347365] rcu-scale: 2 writer-duration: 8 35614
[ 49.347371] rcu-scale: 2 writer-duration: 9 26166
[ 49.347377] rcu-scale: 2 writer-duration: 10 28552
[ 49.347384] rcu-scale: 2 writer-duration: 11 24335
[ 49.347390] rcu-scale: 2 writer-duration: 12 35614
[ 49.347396] rcu-scale: 2 writer-duration: 13 25978
[ 49.347402] rcu-scale: 2 writer-duration: 14 28541
[ 49.347408] rcu-scale: 2 writer-duration: 15 24634
[ 49.347413] rcu-scale: 2 writer-duration: 16 35640
[ 49.347419] rcu-scale: 2 writer-duration: 17 26235
[ 49.347425] rcu-scale: 2 writer-duration: 18 28782
[ 49.347431] rcu-scale: 2 writer-duration: 19 24288
[ 49.347437] rcu-scale: 2 writer-duration: 20 35649
[ 49.347442] rcu-scale: 2 writer-duration: 21 25989
[ 49.347448] rcu-scale: 2 writer-duration: 22 28527
[ 49.347454] rcu-scale: 2 writer-duration: 23 24150
[ 49.347460] rcu-scale: 2 writer-duration: 24 35960
[ 49.347465] rcu-scale: 2 writer-duration: 25 26133
[ 49.347471] rcu-scale: 2 writer-duration: 26 28714
[ 49.347477] rcu-scale: 2 writer-duration: 27 24192
[ 49.347482] rcu-scale: 2 writer-duration: 28 35545
[ 49.347488] rcu-scale: 2 writer-duration: 29 26893
[ 49.347494] rcu-scale: 2 writer-duration: 30 28621
[ 49.347499] rcu-scale: 2 writer-duration: 31 24453
[ 49.347505] rcu-scale: 2 writer-duration: 32 35750
[ 49.347510] rcu-scale: 2 writer-duration: 33 31021
[ 49.347516] rcu-scale: 2 writer-duration: 34 29061
[ 49.347522] rcu-scale: 2 writer-duration: 35 25015
[ 49.347528] rcu-scale: 2 writer-duration: 36 36107
[ 49.347534] rcu-scale: 2 writer-duration: 37 26370
[ 49.347540] rcu-scale: 2 writer-duration: 38 28888
[ 49.347546] rcu-scale: 2 writer-duration: 39 24418
[ 49.347552] rcu-scale: 2 writer-duration: 40 35675
[ 49.347558] rcu-scale: 2 writer-duration: 41 26284
[ 49.347564] rcu-scale: 2 writer-duration: 42 28924
[ 49.347570] rcu-scale: 2 writer-duration: 43 24244
[ 49.347576] rcu-scale: 2 writer-duration: 44 35835
[ 49.347582] rcu-scale: 2 writer-duration: 45 26194
[ 49.347587] rcu-scale: 2 writer-duration: 46 29080
[ 49.347593] rcu-scale: 2 writer-duration: 47 24185
[ 49.347599] rcu-scale: 2 writer-duration: 48 35543
[ 49.347605] rcu-scale: 2 writer-duration: 49 26122
[ 49.347610] rcu-scale: 2 writer-duration: 50 28822
[ 49.347616] rcu-scale: 2 writer-duration: 51 24285
[ 49.347622] rcu-scale: 2 writer-duration: 52 35720
[ 49.347627] rcu-scale: 2 writer-duration: 53 26063
[ 49.347633] rcu-scale: 2 writer-duration: 54 28508
[ 49.347639] rcu-scale: 2 writer-duration: 55 24019
[ 49.347645] rcu-scale: 2 writer-duration: 56 35499
[ 49.347651] rcu-scale: 2 writer-duration: 57 26079
[ 49.347657] rcu-scale: 2 writer-duration: 58 28502
[ 49.347663] rcu-scale: 2 writer-duration: 59 24016
[ 49.347668] rcu-scale: 2 writer-duration: 60 35515
[ 49.347673] rcu-scale: 2 writer-duration: 61 26095
[ 49.347677] rcu-scale: 2 writer-duration: 62 28602
[ 49.347681] rcu-scale: 2 writer-duration: 63 24246
[ 49.347686] rcu-scale: 2 writer-duration: 64 35675
[ 49.347690] rcu-scale: 2 writer-duration: 65 26761
[ 49.347694] rcu-scale: 2 writer-duration: 66 28833
[ 49.347699] rcu-scale: 2 writer-duration: 67 24524
[ 49.347703] rcu-scale: 2 writer-duration: 68 43780
[ 49.347707] rcu-scale: 2 writer-duration: 69 26612
[ 49.347712] rcu-scale: 2 writer-duration: 70 29290
[ 49.347716] rcu-scale: 2 writer-duration: 71 24670
[ 49.347720] rcu-scale: 2 writer-duration: 72 35526
[ 49.347725] rcu-scale: 2 writer-duration: 73 26393
[ 49.347729] rcu-scale: 2 writer-duration: 74 28967
[ 49.347733] rcu-scale: 2 writer-duration: 75 24125
[ 49.347738] rcu-scale: 2 writer-duration: 76 35713
[ 49.347742] rcu-scale: 2 writer-duration: 77 26062
[ 49.347746] rcu-scale: 2 writer-duration: 78 28888
[ 49.347751] rcu-scale: 2 writer-duration: 79 24344
[ 49.347755] rcu-scale: 2 writer-duration: 80 35858
[ 49.347760] rcu-scale: 2 writer-duration: 81 26330
[ 49.347764] rcu-scale: 2 writer-duration: 82 28661
[ 49.347769] rcu-scale: 2 writer-duration: 83 24180
[ 49.347773] rcu-scale: 2 writer-duration: 84 35464
[ 49.347778] rcu-scale: 2 writer-duration: 85 25886
[ 49.347782] rcu-scale: 2 writer-duration: 86 28916
[ 49.347786] rcu-scale: 2 writer-duration: 87 24149
[ 49.347791] rcu-scale: 2 writer-duration: 88 35860
[ 49.347795] rcu-scale: 2 writer-duration: 89 26100
[ 49.347800] rcu-scale: 2 writer-duration: 90 28999
[ 49.347804] rcu-scale: 2 writer-duration: 91 24105
[ 49.347809] rcu-scale: 2 writer-duration: 92 35720
[ 49.347813] rcu-scale: 2 writer-duration: 93 26093
[ 49.347817] rcu-scale: 2 writer-duration: 94 28729
[ 49.347822] rcu-scale: 2 writer-duration: 95 24063
[ 49.347826] rcu-scale: 2 writer-duration: 96 35819
[ 49.347831] rcu-scale: 2 writer-duration: 97 26119
[ 49.347835] rcu-scale: 2 writer-duration: 98 28648
[ 49.347840] rcu-scale: 2 writer-duration: 99 24175
[ 49.347844] rcu-scale: 2 writer-duration: 100 34035
[ 49.349299] rcu-scale: 3 writer-duration: 0 26939
[ 49.351268] rcu-scale: 3 writer-duration: 1 27152
[ 49.351277] rcu-scale: 3 writer-duration: 2 30299
[ 49.351281] rcu-scale: 3 writer-duration: 3 36662
[ 49.351284] rcu-scale: 3 writer-duration: 4 31847
[ 49.351287] rcu-scale: 3 writer-duration: 5 29203
[ 49.351290] rcu-scale: 3 writer-duration: 6 24861
[ 49.351294] rcu-scale: 3 writer-duration: 7 36222
[ 49.351297] rcu-scale: 3 writer-duration: 8 26790
[ 49.351300] rcu-scale: 3 writer-duration: 9 28626
[ 49.351303] rcu-scale: 3 writer-duration: 10 24007
[ 49.351307] rcu-scale: 3 writer-duration: 11 35765
[ 49.351310] rcu-scale: 3 writer-duration: 12 26226
[ 49.351314] rcu-scale: 3 writer-duration: 13 28495
[ 49.351317] rcu-scale: 3 writer-duration: 14 24188
[ 49.351321] rcu-scale: 3 writer-duration: 15 35686
[ 49.351324] rcu-scale: 3 writer-duration: 16 26290
[ 49.351327] rcu-scale: 3 writer-duration: 17 28740
[ 49.351331] rcu-scale: 3 writer-duration: 18 24201
[ 49.351334] rcu-scale: 3 writer-duration: 19 36000
[ 49.351337] rcu-scale: 3 writer-duration: 20 25995
[ 49.351341] rcu-scale: 3 writer-duration: 21 28649
[ 49.351344] rcu-scale: 3 writer-duration: 22 24081
[ 49.351348] rcu-scale: 3 writer-duration: 23 35718
[ 49.351351] rcu-scale: 3 writer-duration: 24 26077
[ 49.351354] rcu-scale: 3 writer-duration: 25 28773
[ 49.351357] rcu-scale: 3 writer-duration: 26 24280
[ 49.351360] rcu-scale: 3 writer-duration: 27 35767
[ 49.351364] rcu-scale: 3 writer-duration: 28 25956
[ 49.351367] rcu-scale: 3 writer-duration: 29 28632
[ 49.351370] rcu-scale: 3 writer-duration: 30 24996
[ 49.351374] rcu-scale: 3 writer-duration: 31 35673
[ 49.351377] rcu-scale: 3 writer-duration: 32 26217
[ 49.351380] rcu-scale: 3 writer-duration: 33 28855
[ 49.351383] rcu-scale: 3 writer-duration: 34 28655
[ 49.351387] rcu-scale: 3 writer-duration: 35 36525
[ 49.351390] rcu-scale: 3 writer-duration: 36 27052
[ 49.351394] rcu-scale: 3 writer-duration: 37 28841
[ 49.351397] rcu-scale: 3 writer-duration: 38 24600
[ 49.351400] rcu-scale: 3 writer-duration: 39 35743
[ 49.351404] rcu-scale: 3 writer-duration: 40 26391
[ 49.351407] rcu-scale: 3 writer-duration: 41 28642
[ 49.351410] rcu-scale: 3 writer-duration: 42 24334
[ 49.351413] rcu-scale: 3 writer-duration: 43 36152
[ 49.351417] rcu-scale: 3 writer-duration: 44 26133
[ 49.351420] rcu-scale: 3 writer-duration: 45 28732
[ 49.351423] rcu-scale: 3 writer-duration: 46 24235
[ 49.351426] rcu-scale: 3 writer-duration: 47 36069
[ 49.351429] rcu-scale: 3 writer-duration: 48 25993
[ 49.351433] rcu-scale: 3 writer-duration: 49 28658
[ 49.351436] rcu-scale: 3 writer-duration: 50 24225
[ 49.351439] rcu-scale: 3 writer-duration: 51 35909
[ 49.351442] rcu-scale: 3 writer-duration: 52 26176
[ 49.351446] rcu-scale: 3 writer-duration: 53 28702
[ 49.351449] rcu-scale: 3 writer-duration: 54 24169
[ 49.351453] rcu-scale: 3 writer-duration: 55 35462
[ 49.351456] rcu-scale: 3 writer-duration: 56 25771
[ 49.351459] rcu-scale: 3 writer-duration: 57 28677
[ 49.351462] rcu-scale: 3 writer-duration: 58 24089
[ 49.351466] rcu-scale: 3 writer-duration: 59 35538
[ 49.351469] rcu-scale: 3 writer-duration: 60 25785
[ 49.351472] rcu-scale: 3 writer-duration: 61 28669
[ 49.351475] rcu-scale: 3 writer-duration: 62 24075
[ 49.351478] rcu-scale: 3 writer-duration: 63 35720
[ 49.351482] rcu-scale: 3 writer-duration: 64 26017
[ 49.351485] rcu-scale: 3 writer-duration: 65 28726
[ 49.351488] rcu-scale: 3 writer-duration: 66 24990
[ 49.351491] rcu-scale: 3 writer-duration: 67 35814
[ 49.351494] rcu-scale: 3 writer-duration: 68 26215
[ 49.351498] rcu-scale: 3 writer-duration: 69 34468
[ 49.351501] rcu-scale: 3 writer-duration: 70 27200
[ 49.351504] rcu-scale: 3 writer-duration: 71 36288
[ 49.351507] rcu-scale: 3 writer-duration: 72 26445
[ 49.351511] rcu-scale: 3 writer-duration: 73 28593
[ 49.351514] rcu-scale: 3 writer-duration: 74 24317
[ 49.351517] rcu-scale: 3 writer-duration: 75 36153
[ 49.351521] rcu-scale: 3 writer-duration: 76 26164
[ 49.351524] rcu-scale: 3 writer-duration: 77 28523
[ 49.351527] rcu-scale: 3 writer-duration: 78 24252
[ 49.351530] rcu-scale: 3 writer-duration: 79 35939
[ 49.351533] rcu-scale: 3 writer-duration: 80 26058
[ 49.351536] rcu-scale: 3 writer-duration: 81 28892
[ 49.351540] rcu-scale: 3 writer-duration: 82 24342
[ 49.351543] rcu-scale: 3 writer-duration: 83 35751
[ 49.351546] rcu-scale: 3 writer-duration: 84 25932
[ 49.351549] rcu-scale: 3 writer-duration: 85 28682
[ 49.351552] rcu-scale: 3 writer-duration: 86 24086
[ 49.351555] rcu-scale: 3 writer-duration: 87 35791
[ 49.351559] rcu-scale: 3 writer-duration: 88 26189
[ 49.351562] rcu-scale: 3 writer-duration: 89 28754
[ 49.351565] rcu-scale: 3 writer-duration: 90 24250
[ 49.351568] rcu-scale: 3 writer-duration: 91 35768
[ 49.351572] rcu-scale: 3 writer-duration: 92 26160
[ 49.351575] rcu-scale: 3 writer-duration: 93 28896
[ 49.351578] rcu-scale: 3 writer-duration: 94 24020
[ 49.351581] rcu-scale: 3 writer-duration: 95 35727
[ 49.351584] rcu-scale: 3 writer-duration: 96 26099
[ 49.351587] rcu-scale: 3 writer-duration: 97 28768
[ 49.351590] rcu-scale: 3 writer-duration: 98 24011
[ 49.351593] rcu-scale: 3 writer-duration: 99 35786
[ 49.351597] rcu-scale: 3 writer-duration: 100 25989
[ 49.353257] rcu-scale: 3 writer-duration: 101 46411
[ 47.850402] rcu-scale: rcu_scale_writer 3 has 100 measurements
[ 47.850427] rcu-scale: rcu_scale_writer 0 has 100 measurements
[ 47.850429] rcu-scale: rcu_scale_writer 1 has 100 measurements
[ 47.850432] rcu-scale: rcu_scale_writer 2 has 100 measurements
[ 47.861470] Dumping ftrace buffer:
[ 47.861486] (ftrace buffer empty)
[ 47.861491] rcu-scale: Test complete
[ 63.668029] rcu-scale:!!! All grace periods expedited, no normal ones to measure!
[ 63.668051] rcu-torture: Stopping rcu_scale_reader task
[ 63.668074] rcu-torture: rcu_scale_reader is stopping
[ 63.668089] rcu-torture: Stopping rcu_scale_reader task
[ 63.668138] rcu-torture: rcu_scale_reader is stopping
[ 63.668151] rcu-torture: Stopping rcu_scale_reader task
[ 63.668191] rcu-torture: rcu_scale_reader is stopping
[ 63.668204] rcu-torture: Stopping rcu_scale_reader task
[ 63.668209] rcu-torture: rcu_scale_reader is stopping
[ 63.668231] rcu-torture: Stopping rcu_scale_writer task
[ 63.668241] rcu-torture: rcu_scale_writer is stopping
[ 63.668249] rcu-scale: writer 0 gps: 102
[ 63.668251] rcu-torture: Stopping rcu_scale_writer task
[ 63.668260] rcu-torture: rcu_scale_writer is stopping
[ 63.668268] rcu-scale: writer 1 gps: 102
[ 63.668270] rcu-torture: Stopping rcu_scale_writer task
[ 63.668279] rcu-torture: rcu_scale_writer is stopping
[ 63.668287] rcu-scale: writer 2 gps: 101
[ 63.668288] rcu-torture: Stopping rcu_scale_writer task
[ 63.668293] rcu-torture: rcu_scale_writer is stopping
[ 63.668310] rcu-scale: writer 3 gps: 102
[ 63.668311] rcu-scale: start: 47784890510 end: 47787901801 duration: 3011291 gps: 407 batches: 2
[ 63.668314] rcu-scale: 0 writer-duration: 0 55830
[ 63.669543] rcu-scale: 0 writer-duration: 1 40421
[ 63.669546] rcu-scale: 0 writer-duration: 2 25268
[ 63.669548] rcu-scale: 0 writer-duration: 3 32703
[ 63.669549] rcu-scale: 0 writer-duration: 4 28624
[ 63.669550] rcu-scale: 0 writer-duration: 5 36967
[ 63.669551] rcu-scale: 0 writer-duration: 6 24686
[ 63.669552] rcu-scale: 0 writer-duration: 7 29598
[ 63.669554] rcu-scale: 0 writer-duration: 8 27026
[ 63.669555] rcu-scale: 0 writer-duration: 9 38789
[ 63.669556] rcu-scale: 0 writer-duration: 10 24398
[ 63.669557] rcu-scale: 0 writer-duration: 11 29562
[ 63.669559] rcu-scale: 0 writer-duration: 12 27348
[ 63.669560] rcu-scale: 0 writer-duration: 13 36315
[ 63.669561] rcu-scale: 0 writer-duration: 14 24233
[ 63.669562] rcu-scale: 0 writer-duration: 15 29035
[ 63.669564] rcu-scale: 0 writer-duration: 16 26754
[ 63.669606] rcu-scale: 0 writer-duration: 17 36272
[ 63.669608] rcu-scale: 0 writer-duration: 18 24237
[ 63.669609] rcu-scale: 0 writer-duration: 19 28834
[ 63.669610] rcu-scale: 0 writer-duration: 20 26993
[ 63.669611] rcu-scale: 0 writer-duration: 21 36432
[ 63.669612] rcu-scale: 0 writer-duration: 22 24178
[ 63.669614] rcu-scale: 0 writer-duration: 23 28838
[ 63.669615] rcu-scale: 0 writer-duration: 24 26733
[ 63.669616] rcu-scale: 0 writer-duration: 25 36087
[ 63.669617] rcu-scale: 0 writer-duration: 26 24237
[ 63.669618] rcu-scale: 0 writer-duration: 27 28656
[ 63.669620] rcu-scale: 0 writer-duration: 28 26921
[ 63.669621] rcu-scale: 0 writer-duration: 29 36423
[ 63.669622] rcu-scale: 0 writer-duration: 30 24201
[ 63.669623] rcu-scale: 0 writer-duration: 31 28748
[ 63.669625] rcu-scale: 0 writer-duration: 32 31925
[ 63.669626] rcu-scale: 0 writer-duration: 33 36673
[ 63.669627] rcu-scale: 0 writer-duration: 34 24630
[ 63.669628] rcu-scale: 0 writer-duration: 35 29033
[ 63.669629] rcu-scale: 0 writer-duration: 36 26761
[ 63.669631] rcu-scale: 0 writer-duration: 37 36495
[ 63.669632] rcu-scale: 0 writer-duration: 38 24209
[ 63.669633] rcu-scale: 0 writer-duration: 39 29468
[ 63.669634] rcu-scale: 0 writer-duration: 40 27137
[ 63.669635] rcu-scale: 0 writer-duration: 41 36193
[ 63.669637] rcu-scale: 0 writer-duration: 42 24531
[ 63.669638] rcu-scale: 0 writer-duration: 43 29113
[ 63.669639] rcu-scale: 0 writer-duration: 44 26915
[ 63.669640] rcu-scale: 0 writer-duration: 45 36394
[ 63.669642] rcu-scale: 0 writer-duration: 46 24247
[ 63.669643] rcu-scale: 0 writer-duration: 47 28823
[ 63.669644] rcu-scale: 0 writer-duration: 48 27029
[ 63.669645] rcu-scale: 0 writer-duration: 49 36042
[ 63.669646] rcu-scale: 0 writer-duration: 50 24164
[ 63.669648] rcu-scale: 0 writer-duration: 51 28972
[ 63.669649] rcu-scale: 0 writer-duration: 52 26858
[ 63.669650] rcu-scale: 0 writer-duration: 53 36319
[ 63.669651] rcu-scale: 0 writer-duration: 54 24233
[ 63.669652] rcu-scale: 0 writer-duration: 55 28986
[ 63.669654] rcu-scale: 0 writer-duration: 56 26895
[ 63.669655] rcu-scale: 0 writer-duration: 57 36213
[ 63.669656] rcu-scale: 0 writer-duration: 58 24156
[ 63.669657] rcu-scale: 0 writer-duration: 59 28859
[ 63.669658] rcu-scale: 0 writer-duration: 60 26917
[ 63.669660] rcu-scale: 0 writer-duration: 61 36174
[ 63.669661] rcu-scale: 0 writer-duration: 62 24237
[ 63.669662] rcu-scale: 0 writer-duration: 63 28873
[ 63.669663] rcu-scale: 0 writer-duration: 64 27045
[ 63.669664] rcu-scale: 0 writer-duration: 65 36167
[ 63.669666] rcu-scale: 0 writer-duration: 66 29556
[ 63.669667] rcu-scale: 0 writer-duration: 67 28900
[ 63.669668] rcu-scale: 0 writer-duration: 68 27072
[ 63.669669] rcu-scale: 0 writer-duration: 69 36733
[ 63.669670] rcu-scale: 0 writer-duration: 70 24327
[ 63.669672] rcu-scale: 0 writer-duration: 71 28842
[ 63.669673] rcu-scale: 0 writer-duration: 72 27027
[ 63.669674] rcu-scale: 0 writer-duration: 73 36228
[ 63.669675] rcu-scale: 0 writer-duration: 74 24137
[ 63.669677] rcu-scale: 0 writer-duration: 75 29924
[ 63.669678] rcu-scale: 0 writer-duration: 76 26775
[ 63.669679] rcu-scale: 0 writer-duration: 77 36300
[ 63.669680] rcu-scale: 0 writer-duration: 78 24547
[ 63.669681] rcu-scale: 0 writer-duration: 79 28852
[ 63.669683] rcu-scale: 0 writer-duration: 80 26945
[ 63.669684] rcu-scale: 0 writer-duration: 81 36127
[ 63.669685] rcu-scale: 0 writer-duration: 82 24373
[ 63.669686] rcu-scale: 0 writer-duration: 83 28853
[ 63.669687] rcu-scale: 0 writer-duration: 84 26652
[ 63.669689] rcu-scale: 0 writer-duration: 85 35952
[ 63.669690] rcu-scale: 0 writer-duration: 86 24379
[ 63.669691] rcu-scale: 0 writer-duration: 87 28948
[ 63.669692] rcu-scale: 0 writer-duration: 88 26958
[ 63.669693] rcu-scale: 0 writer-duration: 89 36302
[ 63.669695] rcu-scale: 0 writer-duration: 90 24181
[ 63.669696] rcu-scale: 0 writer-duration: 91 29120
[ 63.669697] rcu-scale: 0 writer-duration: 92 27032
[ 63.669698] rcu-scale: 0 writer-duration: 93 36083
[ 63.669699] rcu-scale: 0 writer-duration: 94 24248
[ 63.669701] rcu-scale: 0 writer-duration: 95 29205
[ 63.669702] rcu-scale: 0 writer-duration: 96 26577
[ 63.669703] rcu-scale: 0 writer-duration: 97 36187
[ 63.669704] rcu-scale: 0 writer-duration: 98 24472
[ 63.669705] rcu-scale: 0 writer-duration: 99 28883
[ 63.669707] rcu-scale: 0 writer-duration: 100 31999
[ 63.671532] rcu-scale: 0 writer-duration: 101 48271
[ 63.671556] rcu-scale: 1 writer-duration: 0 53537
[ 63.673442] rcu-scale: 1 writer-duration: 1 33311
[ 63.673451] rcu-scale: 1 writer-duration: 2 28033
[ 63.673457] rcu-scale: 1 writer-duration: 3 40200
[ 63.673464] rcu-scale: 1 writer-duration: 4 25479
[ 63.673470] rcu-scale: 1 writer-duration: 5 29908
[ 63.673475] rcu-scale: 1 writer-duration: 6 27336
[ 63.673481] rcu-scale: 1 writer-duration: 7 36948
[ 63.673487] rcu-scale: 1 writer-duration: 8 24216
[ 63.673494] rcu-scale: 1 writer-duration: 9 31639
[ 63.673499] rcu-scale: 1 writer-duration: 10 27175
[ 63.673506] rcu-scale: 1 writer-duration: 11 36695
[ 63.673512] rcu-scale: 1 writer-duration: 12 24609
[ 63.673518] rcu-scale: 1 writer-duration: 13 29140
[ 63.673524] rcu-scale: 1 writer-duration: 14 26955
[ 63.673530] rcu-scale: 1 writer-duration: 15 36122
[ 63.673535] rcu-scale: 1 writer-duration: 16 24236
[ 63.673541] rcu-scale: 1 writer-duration: 17 29137
[ 63.673546] rcu-scale: 1 writer-duration: 18 26766
[ 63.673552] rcu-scale: 1 writer-duration: 19 36207
[ 63.673558] rcu-scale: 1 writer-duration: 20 24215
[ 63.673563] rcu-scale: 1 writer-duration: 21 29239
[ 63.673569] rcu-scale: 1 writer-duration: 22 26866
[ 63.673575] rcu-scale: 1 writer-duration: 23 36054
[ 63.673581] rcu-scale: 1 writer-duration: 24 24012
[ 63.673586] rcu-scale: 1 writer-duration: 25 28914
[ 63.673592] rcu-scale: 1 writer-duration: 26 26891
[ 63.673597] rcu-scale: 1 writer-duration: 27 36102
[ 63.673603] rcu-scale: 1 writer-duration: 28 24039
[ 63.673609] rcu-scale: 1 writer-duration: 29 29125
[ 63.673614] rcu-scale: 1 writer-duration: 30 26926
[ 63.673620] rcu-scale: 1 writer-duration: 31 40754
[ 63.673625] rcu-scale: 1 writer-duration: 32 24519
[ 63.673631] rcu-scale: 1 writer-duration: 33 29397
[ 63.673637] rcu-scale: 1 writer-duration: 34 27267
[ 63.673642] rcu-scale: 1 writer-duration: 35 36358
[ 63.673648] rcu-scale: 1 writer-duration: 36 24007
[ 63.673654] rcu-scale: 1 writer-duration: 37 29326
[ 63.673659] rcu-scale: 1 writer-duration: 38 26854
[ 63.673665] rcu-scale: 1 writer-duration: 39 36625
[ 63.673671] rcu-scale: 1 writer-duration: 40 24437
[ 63.673676] rcu-scale: 1 writer-duration: 41 29314
[ 63.673682] rcu-scale: 1 writer-duration: 42 26955
[ 63.673688] rcu-scale: 1 writer-duration: 43 36415
[ 63.673693] rcu-scale: 1 writer-duration: 44 24098
[ 63.673699] rcu-scale: 1 writer-duration: 45 29216
[ 63.673705] rcu-scale: 1 writer-duration: 46 26904
[ 63.673710] rcu-scale: 1 writer-duration: 47 36286
[ 63.673716] rcu-scale: 1 writer-duration: 48 24127
[ 63.673722] rcu-scale: 1 writer-duration: 49 28887
[ 63.673727] rcu-scale: 1 writer-duration: 50 26894
[ 63.673733] rcu-scale: 1 writer-duration: 51 36145
[ 63.673739] rcu-scale: 1 writer-duration: 52 24238
[ 63.673745] rcu-scale: 1 writer-duration: 53 29136
[ 63.673750] rcu-scale: 1 writer-duration: 54 26895
[ 63.673756] rcu-scale: 1 writer-duration: 55 36087
[ 63.673761] rcu-scale: 1 writer-duration: 56 24215
[ 63.673767] rcu-scale: 1 writer-duration: 57 29034
[ 63.673773] rcu-scale: 1 writer-duration: 58 26817
[ 63.673778] rcu-scale: 1 writer-duration: 59 36069
[ 63.673784] rcu-scale: 1 writer-duration: 60 24255
[ 63.673790] rcu-scale: 1 writer-duration: 61 29004
[ 63.673795] rcu-scale: 1 writer-duration: 62 26878
[ 63.673801] rcu-scale: 1 writer-duration: 63 36186
[ 63.673806] rcu-scale: 1 writer-duration: 64 24355
[ 63.673812] rcu-scale: 1 writer-duration: 65 28884
[ 63.673818] rcu-scale: 1 writer-duration: 66 32061
[ 63.673823] rcu-scale: 1 writer-duration: 67 36261
[ 63.673829] rcu-scale: 1 writer-duration: 68 24453
[ 63.673835] rcu-scale: 1 writer-duration: 69 29508
[ 63.673840] rcu-scale: 1 writer-duration: 70 27004
[ 63.673846] rcu-scale: 1 writer-duration: 71 36240
[ 63.673851] rcu-scale: 1 writer-duration: 72 23958
[ 63.673857] rcu-scale: 1 writer-duration: 73 29276
[ 63.673863] rcu-scale: 1 writer-duration: 74 26793
[ 63.673868] rcu-scale: 1 writer-duration: 75 36779
[ 63.673874] rcu-scale: 1 writer-duration: 76 24462
[ 63.673880] rcu-scale: 1 writer-duration: 77 29156
[ 63.673885] rcu-scale: 1 writer-duration: 78 27128
[ 63.673891] rcu-scale: 1 writer-duration: 79 36182
[ 63.673896] rcu-scale: 1 writer-duration: 80 24073
[ 63.673902] rcu-scale: 1 writer-duration: 81 28955
[ 63.673908] rcu-scale: 1 writer-duration: 82 27009
[ 63.673914] rcu-scale: 1 writer-duration: 83 36169
[ 63.673919] rcu-scale: 1 writer-duration: 84 23906
[ 63.673925] rcu-scale: 1 writer-duration: 85 28946
[ 63.673931] rcu-scale: 1 writer-duration: 86 26866
[ 63.673937] rcu-scale: 1 writer-duration: 87 36249
[ 63.673942] rcu-scale: 1 writer-duration: 88 24096
[ 63.673948] rcu-scale: 1 writer-duration: 89 29290
[ 63.673954] rcu-scale: 1 writer-duration: 90 26844
[ 63.673959] rcu-scale: 1 writer-duration: 91 36353
[ 63.673965] rcu-scale: 1 writer-duration: 92 24166
[ 63.673971] rcu-scale: 1 writer-duration: 93 29072
[ 63.673976] rcu-scale: 1 writer-duration: 94 26862
[ 63.673982] rcu-scale: 1 writer-duration: 95 36348
[ 63.673988] rcu-scale: 1 writer-duration: 96 23907
[ 63.673993] rcu-scale: 1 writer-duration: 97 29122
[ 63.673999] rcu-scale: 1 writer-duration: 98 27072
[ 63.674004] rcu-scale: 1 writer-duration: 99 36188
[ 63.674010] rcu-scale: 1 writer-duration: 100 29125
[ 63.675441] rcu-scale: 1 writer-duration: 101 48063
[ 63.675458] rcu-scale: 2 writer-duration: 0 65839
[ 63.677455] rcu-scale: 2 writer-duration: 1 30102
[ 63.677470] rcu-scale: 2 writer-duration: 2 30481
[ 63.677477] rcu-scale: 2 writer-duration: 3 28129
[ 63.677483] rcu-scale: 2 writer-duration: 4 37897
[ 63.677489] rcu-scale: 2 writer-duration: 5 27420
[ 63.677496] rcu-scale: 2 writer-duration: 6 29475
[ 63.677501] rcu-scale: 2 writer-duration: 7 25128
[ 63.677507] rcu-scale: 2 writer-duration: 8 36449
[ 63.677512] rcu-scale: 2 writer-duration: 9 29093
[ 63.677518] rcu-scale: 2 writer-duration: 10 29369
[ 63.677524] rcu-scale: 2 writer-duration: 11 24938
[ 63.677530] rcu-scale: 2 writer-duration: 12 36634
[ 63.677535] rcu-scale: 2 writer-duration: 13 26628
[ 63.677541] rcu-scale: 2 writer-duration: 14 29116
[ 63.677547] rcu-scale: 2 writer-duration: 15 24320
[ 63.677552] rcu-scale: 2 writer-duration: 16 36351
[ 63.677558] rcu-scale: 2 writer-duration: 17 26613
[ 63.677564] rcu-scale: 2 writer-duration: 18 29039
[ 63.677570] rcu-scale: 2 writer-duration: 19 24353
[ 63.677575] rcu-scale: 2 writer-duration: 20 36335
[ 63.677581] rcu-scale: 2 writer-duration: 21 26706
[ 63.677587] rcu-scale: 2 writer-duration: 22 29200
[ 63.677592] rcu-scale: 2 writer-duration: 23 24211
[ 63.677598] rcu-scale: 2 writer-duration: 24 36036
[ 63.677604] rcu-scale: 2 writer-duration: 25 26464
[ 63.677610] rcu-scale: 2 writer-duration: 26 29181
[ 63.677616] rcu-scale: 2 writer-duration: 27 24021
[ 63.677621] rcu-scale: 2 writer-duration: 28 36169
[ 63.677627] rcu-scale: 2 writer-duration: 29 26697
[ 63.677633] rcu-scale: 2 writer-duration: 30 29213
[ 63.677638] rcu-scale: 2 writer-duration: 31 24181
[ 63.677644] rcu-scale: 2 writer-duration: 32 41287
[ 63.677650] rcu-scale: 2 writer-duration: 33 27016
[ 63.677656] rcu-scale: 2 writer-duration: 34 29455
[ 63.677662] rcu-scale: 2 writer-duration: 35 24571
[ 63.677667] rcu-scale: 2 writer-duration: 36 36144
[ 63.677673] rcu-scale: 2 writer-duration: 37 26673
[ 63.677678] rcu-scale: 2 writer-duration: 38 29031
[ 63.677684] rcu-scale: 2 writer-duration: 39 25127
[ 63.677690] rcu-scale: 2 writer-duration: 40 36453
[ 63.677695] rcu-scale: 2 writer-duration: 41 26455
[ 63.677701] rcu-scale: 2 writer-duration: 42 29460
[ 63.677707] rcu-scale: 2 writer-duration: 43 24698
[ 63.677712] rcu-scale: 2 writer-duration: 44 36093
[ 63.677718] rcu-scale: 2 writer-duration: 45 26573
[ 63.677723] rcu-scale: 2 writer-duration: 46 29157
[ 63.677729] rcu-scale: 2 writer-duration: 47 24356
[ 63.677735] rcu-scale: 2 writer-duration: 48 36504
[ 63.677741] rcu-scale: 2 writer-duration: 49 26157
[ 63.677746] rcu-scale: 2 writer-duration: 50 29103
[ 63.677752] rcu-scale: 2 writer-duration: 51 24577
[ 63.677758] rcu-scale: 2 writer-duration: 52 36218
[ 63.677763] rcu-scale: 2 writer-duration: 53 26553
[ 63.677769] rcu-scale: 2 writer-duration: 54 29224
[ 63.677775] rcu-scale: 2 writer-duration: 55 24290
[ 63.677780] rcu-scale: 2 writer-duration: 56 36322
[ 63.677786] rcu-scale: 2 writer-duration: 57 26509
[ 63.677792] rcu-scale: 2 writer-duration: 58 28951
[ 63.677797] rcu-scale: 2 writer-duration: 59 24316
[ 63.677803] rcu-scale: 2 writer-duration: 60 36402
[ 63.677809] rcu-scale: 2 writer-duration: 61 26453
[ 63.677814] rcu-scale: 2 writer-duration: 62 29023
[ 63.677820] rcu-scale: 2 writer-duration: 63 24538
[ 63.677826] rcu-scale: 2 writer-duration: 64 36279
[ 63.677831] rcu-scale: 2 writer-duration: 65 26277
[ 63.677837] rcu-scale: 2 writer-duration: 66 34442
[ 63.677843] rcu-scale: 2 writer-duration: 67 24564
[ 63.677849] rcu-scale: 2 writer-duration: 68 36429
[ 63.677855] rcu-scale: 2 writer-duration: 69 27018
[ 63.677861] rcu-scale: 2 writer-duration: 70 29207
[ 63.677866] rcu-scale: 2 writer-duration: 71 24488
[ 63.677872] rcu-scale: 2 writer-duration: 72 36187
[ 63.677878] rcu-scale: 2 writer-duration: 73 26564
[ 63.677883] rcu-scale: 2 writer-duration: 74 29127
[ 63.677889] rcu-scale: 2 writer-duration: 75 25213
[ 63.677894] rcu-scale: 2 writer-duration: 76 36202
[ 63.677900] rcu-scale: 2 writer-duration: 77 26709
[ 63.677905] rcu-scale: 2 writer-duration: 78 29292
[ 63.677911] rcu-scale: 2 writer-duration: 79 24338
[ 63.677917] rcu-scale: 2 writer-duration: 80 36322
[ 63.677922] rcu-scale: 2 writer-duration: 81 26289
[ 63.677928] rcu-scale: 2 writer-duration: 82 29386
[ 63.677934] rcu-scale: 2 writer-duration: 83 24300
[ 63.677939] rcu-scale: 2 writer-duration: 84 36011
[ 63.677945] rcu-scale: 2 writer-duration: 85 26219
[ 63.677951] rcu-scale: 2 writer-duration: 86 29250
[ 63.677957] rcu-scale: 2 writer-duration: 87 24484
[ 63.677963] rcu-scale: 2 writer-duration: 88 36254
[ 63.677969] rcu-scale: 2 writer-duration: 89 26631
[ 63.677975] rcu-scale: 2 writer-duration: 90 29105
[ 63.677980] rcu-scale: 2 writer-duration: 91 24502
[ 63.677986] rcu-scale: 2 writer-duration: 92 36409
[ 63.677992] rcu-scale: 2 writer-duration: 93 26414
[ 63.677997] rcu-scale: 2 writer-duration: 94 29130
[ 63.678003] rcu-scale: 2 writer-duration: 95 24586
[ 63.678009] rcu-scale: 2 writer-duration: 96 36159
[ 63.678014] rcu-scale: 2 writer-duration: 97 26263
[ 63.678021] rcu-scale: 2 writer-duration: 98 29479
[ 63.678027] rcu-scale: 2 writer-duration: 99 24352
[ 63.678032] rcu-scale: 2 writer-duration: 100 39478
[ 63.679533] rcu-scale: 3 writer-duration: 0 31164
[ 63.681440] rcu-scale: 3 writer-duration: 1 31690
[ 63.681451] rcu-scale: 3 writer-duration: 2 28398
[ 63.681457] rcu-scale: 3 writer-duration: 3 37524
[ 63.681462] rcu-scale: 3 writer-duration: 4 30331
[ 63.681468] rcu-scale: 3 writer-duration: 5 30588
[ 63.681473] rcu-scale: 3 writer-duration: 6 25430
[ 63.681479] rcu-scale: 3 writer-duration: 7 36628
[ 63.681485] rcu-scale: 3 writer-duration: 8 27195
[ 63.681490] rcu-scale: 3 writer-duration: 9 29267
[ 63.681496] rcu-scale: 3 writer-duration: 10 26967
[ 63.681503] rcu-scale: 3 writer-duration: 11 36610
[ 63.681509] rcu-scale: 3 writer-duration: 12 27017
[ 63.681514] rcu-scale: 3 writer-duration: 13 29493
[ 63.681520] rcu-scale: 3 writer-duration: 14 24647
[ 63.681526] rcu-scale: 3 writer-duration: 15 36099
[ 63.681531] rcu-scale: 3 writer-duration: 16 26574
[ 63.681537] rcu-scale: 3 writer-duration: 17 28995
[ 63.681542] rcu-scale: 3 writer-duration: 18 24627
[ 63.681548] rcu-scale: 3 writer-duration: 19 36099
[ 63.681554] rcu-scale: 3 writer-duration: 20 26580
[ 63.681559] rcu-scale: 3 writer-duration: 21 29133
[ 63.681565] rcu-scale: 3 writer-duration: 22 24684
[ 63.681570] rcu-scale: 3 writer-duration: 23 36089
[ 63.681576] rcu-scale: 3 writer-duration: 24 26508
[ 63.681582] rcu-scale: 3 writer-duration: 25 28821
[ 63.681587] rcu-scale: 3 writer-duration: 26 24382
[ 63.681593] rcu-scale: 3 writer-duration: 27 36151
[ 63.681599] rcu-scale: 3 writer-duration: 28 26336
[ 63.681605] rcu-scale: 3 writer-duration: 29 28946
[ 63.681610] rcu-scale: 3 writer-duration: 30 24735
[ 63.681616] rcu-scale: 3 writer-duration: 31 36237
[ 63.681622] rcu-scale: 3 writer-duration: 32 26239
[ 63.681627] rcu-scale: 3 writer-duration: 33 34242
[ 63.681633] rcu-scale: 3 writer-duration: 34 24917
[ 63.681639] rcu-scale: 3 writer-duration: 35 36459
[ 63.681645] rcu-scale: 3 writer-duration: 36 26774
[ 63.681650] rcu-scale: 3 writer-duration: 37 28911
[ 63.681656] rcu-scale: 3 writer-duration: 38 24797
[ 63.681662] rcu-scale: 3 writer-duration: 39 36099
[ 63.681667] rcu-scale: 3 writer-duration: 40 27206
[ 63.681673] rcu-scale: 3 writer-duration: 41 29201
[ 63.681678] rcu-scale: 3 writer-duration: 42 24676
[ 63.681684] rcu-scale: 3 writer-duration: 43 36260
[ 63.681690] rcu-scale: 3 writer-duration: 44 26859
[ 63.681695] rcu-scale: 3 writer-duration: 45 29087
[ 63.681701] rcu-scale: 3 writer-duration: 46 24531
[ 63.681707] rcu-scale: 3 writer-duration: 47 36144
[ 63.681712] rcu-scale: 3 writer-duration: 48 26415
[ 63.681718] rcu-scale: 3 writer-duration: 49 29230
[ 63.681724] rcu-scale: 3 writer-duration: 50 24371
[ 63.681729] rcu-scale: 3 writer-duration: 51 36045
[ 63.681735] rcu-scale: 3 writer-duration: 52 26707
[ 63.681740] rcu-scale: 3 writer-duration: 53 29079
[ 63.681746] rcu-scale: 3 writer-duration: 54 24622
[ 63.681752] rcu-scale: 3 writer-duration: 55 36202
[ 63.681757] rcu-scale: 3 writer-duration: 56 26489
[ 63.681763] rcu-scale: 3 writer-duration: 57 29016
[ 63.681769] rcu-scale: 3 writer-duration: 58 24632
[ 63.681775] rcu-scale: 3 writer-duration: 59 35968
[ 63.681780] rcu-scale: 3 writer-duration: 60 26480
[ 63.681786] rcu-scale: 3 writer-duration: 61 29171
[ 63.681792] rcu-scale: 3 writer-duration: 62 24542
[ 63.681797] rcu-scale: 3 writer-duration: 63 35972
[ 63.681803] rcu-scale: 3 writer-duration: 64 26459
[ 63.681809] rcu-scale: 3 writer-duration: 65 29248
[ 63.681814] rcu-scale: 3 writer-duration: 66 24403
[ 63.681820] rcu-scale: 3 writer-duration: 67 41353
[ 63.681826] rcu-scale: 3 writer-duration: 68 26759
[ 63.681831] rcu-scale: 3 writer-duration: 69 29167
[ 63.681837] rcu-scale: 3 writer-duration: 70 25051
[ 63.681843] rcu-scale: 3 writer-duration: 71 36298
[ 63.681848] rcu-scale: 3 writer-duration: 72 26557
[ 63.681854] rcu-scale: 3 writer-duration: 73 28940
[ 63.681860] rcu-scale: 3 writer-duration: 74 24726
[ 63.681865] rcu-scale: 3 writer-duration: 75 36109
[ 63.681871] rcu-scale: 3 writer-duration: 76 27216
[ 63.681877] rcu-scale: 3 writer-duration: 77 29170
[ 63.681882] rcu-scale: 3 writer-duration: 78 24606
[ 63.681888] rcu-scale: 3 writer-duration: 79 36459
[ 63.681893] rcu-scale: 3 writer-duration: 80 26470
[ 63.681899] rcu-scale: 3 writer-duration: 81 28939
[ 63.681905] rcu-scale: 3 writer-duration: 82 24758
[ 63.681910] rcu-scale: 3 writer-duration: 83 36139
[ 63.681916] rcu-scale: 3 writer-duration: 84 26535
[ 63.681922] rcu-scale: 3 writer-duration: 85 28824
[ 63.681927] rcu-scale: 3 writer-duration: 86 24272
[ 63.681933] rcu-scale: 3 writer-duration: 87 36187
[ 63.681939] rcu-scale: 3 writer-duration: 88 26565
[ 63.681944] rcu-scale: 3 writer-duration: 89 29050
[ 63.681950] rcu-scale: 3 writer-duration: 90 24686
[ 63.681955] rcu-scale: 3 writer-duration: 91 36216
[ 63.681961] rcu-scale: 3 writer-duration: 92 26589
[ 63.681967] rcu-scale: 3 writer-duration: 93 29076
[ 63.681973] rcu-scale: 3 writer-duration: 94 24683
[ 63.681978] rcu-scale: 3 writer-duration: 95 36081
[ 63.681984] rcu-scale: 3 writer-duration: 96 26710
[ 63.681989] rcu-scale: 3 writer-duration: 97 28881
[ 63.681995] rcu-scale: 3 writer-duration: 98 24494
[ 63.682001] rcu-scale: 3 writer-duration: 99 36297
[ 63.682006] rcu-scale: 3 writer-duration: 100 26487
[ 63.683440] rcu-scale: 3 writer-duration: 101 48416