Re: [PATCH 2/2] tracing: Add a verifier to check string pointers for trace events

[Sean Christopherson]

On Fri, Jun 04, 2021, Steven Rostedt wrote:
> On Fri, 4 Jun 2021 22:28:30 -0400
> Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
> 
> > What you described to me does not sound like a false positive, and
> > converting to __string() is not a workaround but an actual fix, and
> > would also need to be backported to stable.
> 
> If the event you are talking about is kvm_nested_vmenter_failed, which
> records a pointer to the message, then, yes, that is unsafe and buggy.

Yep, that's the one.

> If that event is defined by the kvm module, and used in kvm_intel. If
> you have this:
> 
> 	# echo 1 > /sys/kernel/tracing/events/kvm/kvm_nested_vmenter_failed
> 
> 	# do_whatever_causes_that_to_trigger
> 
> 	# rmmod kvm_intel
> 
> 	# cat trace
> 
> Then that pointer to the string will point to some random memory.
> Before this patch, it could even possibly crash the kernel!

I assumed that was the gist of the unsafe string detection, but the module core
data exemption confused me.  I take it that the tracepoint itself goes away if
the module is unloaded?  I.e. it's safe for a module to pass a constant string to
its own tracepoints, but not to tracepoints defined elsewhere?  The comment
above tracepoint_string() seems to confirm this.

> There's two fixes you can do with this. One is to covert that to use
> __string, the other is to do what RCU does, and use the
> tracepoint_string() functionality.
> 
> RCU has:
> 
> 	#define TPS(x)	tracepoint_string(x)
> 
> And wrap all strings with that TPS(), like in nested_vmx_reflect_vmexit():
> 
>         if (unlikely(vmx->fail)) {
>                 trace_kvm_nested_vmenter_failed(
> -                       "hardware VM-instruction error: ",
> +                       TPS("hardware VM-instruction error: "),
>                         vmcs_read32(VM_INSTRUCTION_ERROR));
>                 exit_intr_info = 0;
>                 exit_qual = 0;
>                 goto reflect_vmexit;
>         }
> 
> What the tracepoint_string does is to save the string into core kernel
> memory (there's logic to use the same string if it is already
> available), and it wont free it when the module is unloaded.

This doesn't appear to be correct.  Were you thinking of something else?

Unless I'm misreading the code and section output, tracepoint_string() just saves
a pointer to the string into a dedicated section, it doesn't magically hoist the
string itself into the kernel proper.  And I can't see how that would work, e.g.
if the module is compiled and linked independent from the core kernel.

And the comment above the macro strongly suggests that persistence needs to be
guaranteed by the entity using tracepoint_string().  Testing bears this out, e.g.
the WARN and UNSAFE_MEMORY errors still happen when using tracepoint_string().

 * The @str used must be a constant string and persistent as it would not
 * make sense to show a string that no longer exists. But it is still fine
 * to be used with modules, because when modules are unloaded, if they
 * had tracepoints, the ring buffers are cleared too. As long as the string
 * does not change during the life of the module, it is fine to use
 * tracepoint_string() within a module.
 */
#define tracepoint_string(str)						\
	({								\
		static const char *___tp_str __tracepoint_string = str; \
		___tp_str;						\
	})


Thanks for the help!

> This makes the string safe to use by the trace event directly.
> 
> Not only is the TPS safe, it also allows userspace tools to know what
> the string is, as it is exported in printk_formats. Otherwise trace-cmd
> and perf will only display a pointer hex value as it has no idea what
> its pointing to (both TPS and __string fixes this).