Re: [PATCH v6 1/3] x86/fpu: Measure the Latency of XSAVES and XRSTORS

[Yi Sun]

On 02.09.2023 12:09, Andi Kleen wrote:
> > Instead of adding overhead to the regular FPU context saving/restoring code
> > paths, could you add a helper function that has tracing code included, but
> > which isn't otherwise used - and leave the regular code with no tracing
> > overhead?

> > This puts a bit of a long-term maintenance focus on making sure that the
> > traced functionality won't bitrot, but I'd say that's preferable to adding
> > tracing overhead.
>
> Or just use PT
>
> % sudo perf record --kcore -e intel_pt/cyc=1,cyc_thresh=1/k --filter 'filter save_fpregs_to_fpstate' -a sleep 5
> % sudo perf script --insn-trace --xed -F -comm,-tid,-dso,-sym,-symoff,+ipc
> [000] 677203.751913565:  ffffffffa7046230               nopw  %ax, (%rax)
> [000] 677203.751913565:  ffffffffa7046234               nopl  %eax, (%rax,%rax,1)
> [000] 677203.751913565:  ffffffffa7046239               mov %rdi, %rcx
> [000] 677203.751913565:  ffffffffa704623c               nopl  %eax, (%rax,%rax,1)
> [000] 677203.751913565:  ffffffffa7046241               movq
> 0x10(%rdi), %rsi
> [000] 677203.751913565:  ffffffffa7046245               movq  0x8(%rsi), %rax
> [000] 677203.751913565:  ffffffffa7046249               leaq 0x40(%rsi), %rdi
> [000] 677203.751913565:  ffffffffa704624d               mov %rax, %rdx
> [000] 677203.751913565:  ffffffffa7046250               shr $0x20, %rdx
> [000] 677203.751913565:  ffffffffa7046254               xsaves64 (%rdi)
> [000] 677203.751913565:  ffffffffa7046258               xor %edi, %edi
> [000] 677203.751913565:  ffffffffa704625a               movq 0x10(%rcx), %rax
> [000] 677203.751913565:  ffffffffa704625e               testb  $0xc0, 0x240(%rax)
> [000] 677203.751913636:  ffffffffa7046265               jz 0xffffffffa7046285    IPC: 0.16 (14/85)
> ...
>
>
> So it took 85 cycles here.
>
> (it includes a few extra instructions, but I bet they're less than what
> ftrace adds. This example is for XSAVE, but can be similarly extended
> for XRSTOR)
Hi Andi,
Thank you for your guidance on Intel PT.

I recall that we have discussed this topic via email before.
I have compared the two methods that calculate the latency:
 1. Calculate using perf-intel-pt with functions filter.
 2. Calculate the tsc delta explicitly in kernel, and dump the delta by a
 single trace point as what this patch does.

My findings are:
 1. Intel-pt is the most accurate method, but it's likely just a one-time
 exercise because 'filter with function' requires rebuilding the kernel
 and changing the definition of functions 'os_xsave' and 'os_xrstor' into
 'noinline' instead of 'inline'.
 2. I collected the latency data with the two methods, and the method in
 this patch can achieve results that are close to those with intel-pt.
 And it only introduces a negligible impact on the performance when the
 trace is disabled, as I explained to Ingo earlier.
 
Hope this clarifies my approach. We're using this patch set to do tests
on Intel's new brand chipsets.

Thanks
   --Sun, Yi