Tracing text poke / kernel self-modifying code (Was: Re: [RFC v2 0/6] x86: dynamic indirect branch promotion)

[Adrian Hunter]

On 9/01/19 12:35 PM, Peter Zijlstra wrote:
> On Tue, Jan 08, 2019 at 12:47:42PM -0800, Nadav Amit wrote:
> 
>> A general solution is more complicated, however, due to the racy nature of
>> cross-modifying code. There would need to be TSC recording of the time
>> before the modifications start and after they are done.
>>
>> BTW: I am not sure that static-keys are much better. Their change also
>> affects the control flow, and they do affect the control flow.
> 
> Any text_poke() user is a problem; which is why I suggested a
> PERF_RECORD_TEXT_POKE that emits the new instruction. Such records are
> timestamped and can be correlated to the trace.
> 
> As to the racy nature of text_poke, yes, this is a wee bit tricky and
> might need some care. I _think_ we can make it work, but I'm not 100%
> sure on exactly how PT works, but something like:
> 
>  - write INT3 byte
>  - IPI-SYNC
> 
> and ensure the poke_handler preserves the existing control flow (which
> it currently does not, but should be possible).
> 
>  - emit RECORD_TEXT_POKE with the new instruction
> 
> at this point the actual control flow will be through the INT3 and
> handler and not hit the actual instruction, so the actual state is
> irrelevant.
> 
>  - write instruction tail
>  - IPI-SYNC
>  - write first byte
>  - IPI-SYNC
> 
> And at this point we start using the new instruction, but this is after
> the timestamp from the RECORD_TEXT_POKE event and decoding should work
> just fine.
> 

Presumably the IPI-SYNC does not guarantee that other CPUs will not already
have seen the change.  In that case, it is not possible to provide a
timestamp before which all CPUs executed the old code, and after which all
CPUs execute the new code.

So we need 2 events: one before and one after the text poke.  Then it will
be up to the tools to figure out which code path was taken in that time
interval. e.g.

	1. emit RECORD_TEXT_POKE
		flags:	BEFORE (timestamp is before change)
		method:	INT3 (INT3 method used to change code)
		ip
		code size
		code bytes before
		code bytes after

	2. text poke

	3. emit RECORD_TEXT_POKE
		flags:	AFTER (timestamp is after change)
		method:	INT3 (INT3 method used to change code)
		ip
		code size
		code bytes before
		code bytes after

Thoughts?