Re: Instrumentation and RCU

[Mathieu Desnoyers]

----- On Mar 10, 2020, at 12:48 PM, Thomas Gleixner tglx@xxxxxxxxxxxxx wrote:

> Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx> writes:
[...]
> See the entry series I'm working on. Aside of moving all this nonsense
> into C-code it splits lockdep and tracing so it looks like this:
> 
>            lockdep_hardirqs_off();
>            user_exit_irqsoff();
>            __trace_hardirqs_off();
> 
> The latter uses regular RCU and not the scru/rcu_irq dance.
> 

Awesome :)

> 
>>>> If a tracer recurses, or if a tracer attempts to trace another tracer, the
>>>> instrumentation would break the recursion chain by preventing instrumentation
>>>> from firing. If we end up caring about tracers tracing other tracers, we could
>>>> have one distinct flag per tracer and let each tracer break the recursion chain.
>>>>
>>>> Having this flag per kernel stack rather than per CPU or per thread would
>>>> allow tracing of nested interrupt handlers (and NMIs), but would break
>>>> call chains both within the same stack or going through a trap. I think
>>>> it could be a nice complementary safety net to handle mishaps in a non-fatal
>>>> way.
>>> 
>>> That works as long as none of this uses breakpoint based patching to
>>> dynamically disable/enable stuff.
>>
>> I'm clearly missing something here. I was expecting the "in_tracing" flag trick
>> to be able to fix the breakpoint recursion issue. What is the problem I'm
>> missing
>> here ?
> 
> How do you "fix" that when you can't reach the tracepoint because you
> trip over a breakpoint and then while trying to fixup that stuff you hit
> another one?

I may still be missing something, but if the fixup code (AFAIU the code performing
the out-of-line single-stepping of the original instruction) belongs to a section
hidden from instrumentation, it should not be an issue.

The basic idea would be, e.g. pseudo-code for int3:

<int3>  <---- in section which cannot be instrumented
if (recursion_ctx->in_tracer) {
   single-step original instruction
   iret
}
[...] prepare stuff
recursion_ctx->in_tracer = true;
instr_allowed()

call external kernel functions (which can be instrumented)

instr_disallowed()
recursion_ctx->in_tracer = false;
single-step original instruction
iret

The purpose of the "in_tracer" flag is to protect whatever is done within external
kernel functions (which can be instrumented) from triggering tracer recursion. It
needs to be combined with hiding of early/late low-level entry/exit functions from
instrumentation (as you propose) to work.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com