Re: [PATCH v7 0/4] perf: add support for profiling jitted code

[Stephane Eranian]

Brendan,

On Thu, Oct 1, 2015 at 3:45 PM, Brendan Gregg <brendan.d.gregg@xxxxxxxxx> wrote:
> G'Day,
>
> On Wed, Sep 30, 2015 at 11:45 PM, Stephane Eranian <eranian@xxxxxxxxxx> wrote:
>>
>> This patch series extends perf record/report/annotate to enable
>> profiling of jitted (just-in-time compiled) code. The current
>> perf tool provides very limited support for profiling jitted
>> code for some runtime environments. But the support is experimental
>> and cannot be used in complex environments. It relies on files
>> in /tmp, for instance. It does not support annotate mode or
>> rejitted code.
>>
>> This patch series adds a better way of profiling jitted code
>> with the following advantages:
>>    - support any jitted code environment (some with modifications)
>>    - support Java runtime with JVMTI interface with no modifications
>>    - provides a portable JVMTI agent library
>>    - known to support V8 runtime
>>    - known to support DART runtime
>>    - supports code rejitting and code movements
>>    - no files in /tmp
>>    - meta-data file is unique to each run
>>    - no changes to perf report/annotate
>>    - support per-thread and system-wide profiling
>>    - support monitoring of multiple simultaneous Jit runtimes
>>    - source level view in perf annotate
>>
>> The support is based on cooperation with the runtime. For Java runtimes,
>> supporting the JVMTI interface, there is no change necessary. For other
>> runtimes, modifications are necessary to emit the meta-data to support
>> symbolization, annotation, source lines correlation of the samples.
>> Those modifications are relatively straighforward, some have been
>> implemented in V8 and DART.
>>
>> The jit environment emits a binary dump file which contains the jitted
>> code (in raw format) and meta-data describing the mapping of functions.
>> The binary format is documented in the jitdump.h header file. It is
>> adapted from the OProfile jitdump format.
>
> While this is impressive work, I don't think I'd use it very much, and
> I wouldn't encourage others too either. Is it right that this approach
> needs to be turned on from runtime start, and will constantly emit
> timestamped JIT records? I'd use that as a backup for existing
> techniques for perf_events and jitted runtimes.
>
This boils down that when does the JIT runtime emit the jitdump data?
In the case of openJDK and given how I wrote the agent, it needs to
run from start to finish.  In order for perf to have full visibility, it needs
to get info about all the code that has been jitted so far. This could be
done after start if there was somehow a protocol to handle this in the
runtime, like a signal. It would just need to dump the current status, including
all the code. The rest of the  support would work. In other words, if there was
a way to signal to the runtime that it is being monitored and that it needs to
dump its state, then everything would work. To avoid generating more dumps,
the runtime would also have to be informed that monitoring has stopped.


> Right now we (Netflix) can profile Java in production using
> perf_events, using an on-demand symbol dump: the perf-map-agent
> JVMTI[1]. This agent used to be loaded on startup, like this patchset,
> which cost overhead: CPU, filesystem I/O, storage. The problem was
> that we didn't know which of the tens of thousands of instances we'd
> want to profile, so loading an always-on JIT symbol dumper on all
> instances would cost resources. With on-demand, we only dump symbols
> on the few instances needed. On-demand has caveats, of course, and
> symbols can change during the profile. But so far that's not been a
> problem.
>
I am guessing that for what you are doing getting the jitted code symbol names
are sufficient. This patch series is not really needed for this, you
already have the
/tmp/perf-map approach. This is mostly for the user program
developers. This series
addresses the  needs of the other category, namely the jit compiler and runtime
developers who need to study the quality of the generated code. And there, they
need to see the profile of the jitted native code. This is the major
value-add of this
series. I would also add the support for re-jitting and code movements.

> So this seems like a lot of changes for what won't be our primary way
> of using perf_events on Java or Node.js, which we currently can
> profile with perf_events effectively. So long as we're aware of that.
>
The changes to perf are small. There are no change to perf record, report,
annotate. just the injection code with the jitdump reader and elf creation.

> Brendan
>
> [1] http://techblog.netflix.com/2015/07/java-in-flames.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/