Re: cpu-freq: running the perf increases the data rate?

[Subhashini Rao Beerisetty]

On Fri, Aug 28, 2020 at 6:04 PM Artem Bityutskiy <dedekind1@xxxxxxxxx> wrote:
>
> On Thu, 2020-08-27 at 22:25 +0530, Subhashini Rao Beerisetty wrote:
> > I have an application which finds the data rate over the PCIe
> > interface. I’m getting the lesser data rate in one of my Linux X86
> > systems.
>
> Some more description, may be? Do you have a PCIe device reading one
> RAM buffer and then writing to another RAM buffer? Or does it generate
> dome data and writes them to a RAM buffer? Presumably it uses DMA? How
> much is the CPU involved into the process? Are we talking about
> transferring few kilobytes or gigabytes?
Thanks a lot for your help and reply.
Regarding hardware setup, Xilinx PCIe FPGA endpoint is connected to
HOST CPU via PCIe bus.
Xilinx PCIe FPGA endpoint has the DMA_REF block and it provides a
mechanism to DMA transfer data at the maximum rate between host CPU
memory and a FIFO in the DMA-REF block.
The host software sets up some data in it’s memory, it then transfers
the data to the DMA-REF’s FIFO and then reads it back into a different
location in its host memory. This is repeated in a loop.  There is a
register in the DMA-REF block that gives an indication of transfer
speed.


>
> > When I change the scaling_governor from "powersave" to "performance"
> > mode for each CPU, then there is slight improvement in the PCIe data
> > rate.
>
> Definitely this makes your CPU(s) run at max speed, but depending on
> platform and settings, this may also affect C-states. Are the CPU(s)
> generally idle while you measure, or busy (involved into the test)? You
> could run 'turbostat' while measuring the bandwidth, to get some CPU
> statistics (e.g., do C-states happen during the PCI test, how busy are
> the CPUs).
>
> > Parallely I started profiling the workload with perf. Whenever I start
> > running the profile command “perf stat -a -d -p <PID>” surprisingly
> > the application resulted in excellent data rate over PCIe, but when I
> > kill the perf command again PCIe data rate drops. I am really confused
> > about this behavior.Any clues from this behaviour?
>
> Well, one possible reason that comes to mind - you get rid of C-states
> when you rung perf, and this increases the PCI bandwidth. You can just
> try disabling C-states (there are sysfs knobs) and check it out.
> Turbostat could be useful to check for this (with and without perf, run
> 'turbostat sleep 10' or something like this (measure for 10 seconds in
> this example), do this while running your PCI test.
Disabling the C-states improved the throughput a lot, thanks a lot for
pointing this out. Could you please give some more explanation on how
disabling C-states improved the throughput?
As you suggested I collected and attached the turbostat log with and
without perf while running the PCIe test.
In my system, only 'performance\powersave' are listed in
scaling_available_governors. Rest other governors
"userspace\ondemand\schedutil" are not listed in available_goverors.
What might be the reason for this?

>
> But I am really just guessing here, I do not know enough about your
> test and the system (e.g., "a Linux x86" system can be so many things,
> like Intel or AMD server or a mobile device)…
It's an Intel Atom processor.
>
>
turbostat version 17.06.23 - Len Brown <lenb@xxxxxxxxxx>
CPUID(0): GenuineIntel 11 CPUID levels; family:model:stepping 0x6:37:9 (6:55:9)
CPUID(1): SSE3 MONITOR - EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, No-TURBO, DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
SLM BCLK: 83.3 Mhz
cpu2: MSR_CC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-CC6-Demotion)
cpu2: MSR_MC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-MC6-Demotion)
RAPL: 4581 sec. Joule Counter Range, at 30 Watts
cpu2: MSR_PLATFORM_INFO: 0x60000001700
6 * 83.3 = 499.8 MHz max efficiency frequency
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x00000000 (C1E auto-promotion: DISabled)
cpu2: MSR_ATOM_CORE_RATIOS: 0x00170602
2 * 83.3 = 166.6 MHz minimum operating frequency
6 * 83.3 = 499.8 MHz low frequency mode (LFM)
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_ATOM_CORE_TURBO_RATIOS: 0x17171717
23 * 83.3 = 1915.9 MHz max turbo 4 active cores
23 * 83.3 = 1915.9 MHz max turbo 3 active cores
23 * 83.3 = 1915.9 MHz max turbo 2 active cores
23 * 83.3 = 1915.9 MHz max turbo 1 active core
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x0017000f (UNlocked: pkg-cstate-limit=15: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C6N: MWAIT 0x58
cpu2: C6S: MWAIT 0x52
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: performance
cpufreq intel_pstate no_turbo: 1
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x00000505 (0.031250 Watts, 0.000032 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x003880fa (UNlocked)
cpu0: PKG Limit #1: ENabled (7.812500 Watts, 262144.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: DISabled (0.000000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POWER_LIMIT: 0x00020000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.001953 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x006e0000 (110 C)
40.004417 sec
Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	C1	C6N	C6S	C1%	C6N%	C6S%	CPU%c1	CPU%c6	Mod%c6	CoreTmp	GFX%rc6	Pkg%pc6	PkgWatt	CorWatt
-	-	748	39.33	1901	1917	458560	0	1566280	1889	7775	43.07	0.29	18.19	42.23	17.85	13.56	29	100.00	11.80	1.14	0.95
0	0	643	33.89	1896	1917	33647	0	405339	492	1950	48.44	0.31	18.28	47.57	17.69	13.32	28	100.00	11.80	1.14	0.95
1	1	1064	55.78	1906	1917	358521	0	449875	25	1910	27.37	0.03	17.83	26.41	17.34	13.32	29
2	2	647	34.14	1895	1917	33821	0	359155	517	1940	48.21	0.29	18.20	47.43	17.86	13.81	28
3	3	638	33.52	1902	1917	32571	0	351911	855	1975	48.27	0.54	18.47	47.51	18.51	13.81	28
turbostat version 17.06.23 - Len Brown <lenb@xxxxxxxxxx>
CPUID(0): GenuineIntel 11 CPUID levels; family:model:stepping 0x6:37:9 (6:55:9)
CPUID(1): SSE3 MONITOR - EIST TM2 TSC MSR ACPI-TM TM
CPUID(6): APERF, No-TURBO, DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, EPB
cpu2: MSR_IA32_MISC_ENABLE: 0x00850089 (TCC EIST No-MWAIT PREFETCH TURBO)
CPUID(7): No-SGX
SLM BCLK: 83.3 Mhz
cpu2: MSR_CC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-CC6-Demotion)
cpu2: MSR_MC6_DEMOTION_POLICY_CONFIG: 0x00000000 (DISable-MC6-Demotion)
RAPL: 4581 sec. Joule Counter Range, at 30 Watts
cpu2: MSR_PLATFORM_INFO: 0x60000001700
6 * 83.3 = 499.8 MHz max efficiency frequency
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_IA32_POWER_CTL: 0x00000000 (C1E auto-promotion: DISabled)
cpu2: MSR_ATOM_CORE_RATIOS: 0x00170602
2 * 83.3 = 166.6 MHz minimum operating frequency
6 * 83.3 = 499.8 MHz low frequency mode (LFM)
23 * 83.3 = 1915.9 MHz base frequency
cpu2: MSR_ATOM_CORE_TURBO_RATIOS: 0x17171717
23 * 83.3 = 1915.9 MHz max turbo 4 active cores
23 * 83.3 = 1915.9 MHz max turbo 3 active cores
23 * 83.3 = 1915.9 MHz max turbo 2 active cores
23 * 83.3 = 1915.9 MHz max turbo 1 active core
cpu2: MSR_PKG_CST_CONFIG_CONTROL: 0x0017000f (UNlocked: pkg-cstate-limit=15: pc7)
cpu2: POLL: CPUIDLE CORE POLL IDLE
cpu2: C1: MWAIT 0x00
cpu2: C6N: MWAIT 0x58
cpu2: C6S: MWAIT 0x52
cpu2: cpufreq driver: intel_pstate
cpu2: cpufreq governor: performance
cpufreq intel_pstate no_turbo: 1
cpu0: MSR_IA32_ENERGY_PERF_BIAS: 0x00000006 (balanced)
cpu0: MSR_RAPL_POWER_UNIT: 0x00000505 (0.031250 Watts, 0.000032 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x003880fa (UNlocked)
cpu0: PKG Limit #1: ENabled (7.812500 Watts, 262144.000000 sec, clamp DISabled)
cpu0: PKG Limit #2: DISabled (0.000000 Watts, 0.000977* sec, clamp DISabled)
cpu0: MSR_PP0_POWER_LIMIT: 0x00020000 (UNlocked)
cpu0: Cores Limit: DISabled (0.000000 Watts, 0.001953 sec, clamp DISabled)
cpu0: MSR_IA32_TEMPERATURE_TARGET: 0x006e0000 (110 C)
40.003807 sec
Core	CPU	Avg_MHz	Busy%	Bzy_MHz	TSC_MHz	IRQ	SMI	C1	C6N	C6S	C1%	C6N%	C6S%	CPU%c1	CPU%c6	Mod%c6	CoreTmp	GFX%rc6	Pkg%pc6	PkgWatt	CorWatt
-	-	522	27.23	1915	1917	285074	0	784132	86336	168235	31.17	13.41	28.89	30.75	26.97	3.75	30	100.00	3.60	1.12	0.94
0	0	412	21.50	1915	1917	11229	0	155840	13229	69746	24.68	7.95	46.57	24.35	35.28	3.80	29	100.00	3.60	1.12	0.94
1	1	764	39.91	1914	1917	242649	0	269925	25179	32418	26.95	13.44	20.52	26.38	19.26	3.80	29
2	2	476	24.82	1915	1917	16549	0	224377	27839	1269	48.85	21.52	5.44	48.36	19.04	3.70	30
3	3	435	22.70	1915	1917	14647	0	133990	20089	64802	24.19	10.74	43.05	23.90	34.28	3.70	30