Re: rcu-bh design

[Joel Fernandes]

On 05/04/2018 03:49 PM, Paul E. McKenney wrote:
> > Yes just one more ;-). I am trying to write a 'probetorture' test inspired
> > by RCU torture that whacks the tracepoints in various scenarios. One of the
> > things I want to do is verify the RCU callbacks are queued and secondly,
> > they are executed. Just to verify that the garbage collect was done and
> > we're not leaking the function probe table (not that I don't have
> > confidence in the chained callback technique which you mentioned, but it
> > would be nice to assure this mechanism is working for tracepoints).
> >
> > Is there a way to detect this given a reference to srcu_struct? Mathieu and
> > we were chatting about srcu_barrier which is cool but that just tells me
> > that if there was a callback queued, it would have executed after the
> > readers were done. But not whether something was queued.
>
> Suppose that you are queuing an RCU callback that in turn queues an SRCU
> callback on my_srcu_struct, like this:
>
> 	void my_rcu_callback(struct rcu_head *rhp)
> 	{
> 		p = container_of(rhp, struct my_struct, my_rcu_head);
>
> 		free_it_up_or_down(p);
> 	}
>
> 	void my_srcu_callback(struct rcu_head *rhp)
> 	{
> 		call_rcu(rhp, my_rcu_callback);
> 	}
>
> 	call_srcu(&my_srcu_struct, &p->my_rcu_head, my_srcu_callback);
>
> Then to make sure that any previously submitted callback has been fully
> processed, you do this:
>
> 	rcu_barrier();
> 	srcu_barrier(&my_srcu_struct);
>
> Of course if you queue in the opposite order, like this:
>
> 	void my_srcu_callback(struct rcu_head *rhp)
> 	{
> 		p = container_of(rhp, struct my_struct, my_rcu_head);
>
> 		free_it_up_or_down(p);
> 	}
>
> 	void my_rcu_callback(struct rcu_head *rhp)
> 	{
> 		call_srcu(&my_srcu_struct, &p->my_rcu_head, my_srcu_callback);
> 	}
>
> 	call_rcu(rhp, my_rcu_callback);
>
> Then you must wait in the opposite order:
>
> 	rcu_barrier();
> 	srcu_barrier(&my_srcu_struct);
>
> Either way, the trick is that the first *_barrier() call cannot return
> until after all previous callbacks have executed, which means that by that
> time the callback is enqueued for the other flavor of {S,}RCU.  So the
> second *_barrier() call must wait for the callback to be completely done,
> through both flavors of {S,}RCU.
>
> So after executing the pair of *_barrier() calls, you know that the
> callback is no longer queued.
>
> Does that cover it, or am I missing a turn in here somewhere?

Yes, that covers some of it. Thanks a lot. Btw, I was also thinking I 
want to check that srcu received a callback queuing request as well (not 
just completion but also queuing).

Say that the tracepoint code is buggy and for some reason a call_srcu 
wasn't done. For example, say the hypothetical bug I'm taking about is 
in tracepoint_remove_func which called the rcu_assign_pointer, but 
didn't call release_probes:

diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c
index d0639d917899..f54cb358f451 100644
--- a/kernel/tracepoint.c
+++ b/kernel/tracepoint.c
@@ -216,7 +216,6 @@ static int tracepoint_add_func(struct tracepoint *tp,
        rcu_assign_pointer(tp->funcs, tp_funcs);
        if (!static_key_enabled(&tp->key))
                static_key_slow_inc(&tp->key);
-       release_probes(old);
        return 0;
 }
---
I want to catching this from the test code. If I did the following, it 
would be insufficient then:

trace_probe_register(...);
srcu_barrier(&my_srcu_struct);  // only tells me any queued calls
				// are done but not that something
				// was queued

I was seeing if I could use my_srcu_struct::completed for that. but I'm 
not sure if that'll work (or that its legal to access it directly - but 
hey this is just test code! :P).

thanks,

 - Joel