Re: [PATCH v1 0/4] perf: enable compression of record mode trace to save storage space

[Jiri Olsa]

On Mon, Jan 14, 2019 at 11:43:31AM +0300, Alexey Budankov wrote:
> Hi,
> On 09.01.2019 20:28, Jiri Olsa wrote:
> > On Mon, Dec 24, 2018 at 04:21:33PM +0300, Alexey Budankov wrote:
> >>
> >> The patch set implements runtime record trace compression accompanied by 
> >> trace file decompression implemented in the tool report mode. Zstandard 
> >> library API [1] is used for compression/decompression of data that come 
> >> from perf_events kernel data buffers.
> >>
> >> Realized -z,--compression_level=n option provides ~3-5x avg. trace file 
> >> size reduction on the tested workloads what significantly saves user's 
> >> storage space on larger server systems where trace file size can easily 
> >> reach several tens or even hundreds of GiBs, especially when profiling 
> >> with stacks for later dwarf unwinding, context-switches tracing and etc.
> >>
> >> The option is effective jointly with asynchronous trace writing because 
> >> compression requires auxiliary memory buffers to operate on and memory 
> >> buffers for asynchronous trace writing serve that purpose.
> > 
> > I dont like that it's onlt for aio only, I can't really see why it's
> 
> For serial streaming, on CPU bound codes, under full system utilization it 
> can induce more runtime overhead and increase data loss because amount of 
> code on performance critical path grows, of course size of written data 
> reduces but still. Feeding kernel buffer content by user space code to a 
> syscall is extended with intermediate copying to user space memory with 
> doing some math on it in the middle.
> 
> > a problem for normal data.. can't we just have one layer before and
> > stream the data to the compress function instead of the file (or aio
> > buffers).. and that compress functions would spit out 64K size COMPRESSED
> > events, which would go to file (or aio buffers)
> 
> It is already almost like that. Compression could be bridged using AIO 
> buffers but then still streamed to file serially using record__pushfn() 
> and that would make some sense for moderate profiling cases on systems 
> without AIO support and trace streaming based on it.
> 
> > 
> > the report side would process them (decompress) on the session layer
> > before the tool callbacks are called
> 
> It is already pretty similar to that.

hum, AFAICS you do that in report code not in on the session layer

jirka