Re: [RFC/RFT][PATCH v2] cpuidle: New timer events oriented governor for tickless systems

[Giovanni Gherdovich]

On Fri, 2018-11-02 at 08:39 -0700, Doug Smythies wrote:
>Â
> I have been testing this V2 against a baseline that includes all
> of the pending menu patches. My baseline kernel is somewhere
> after 4.19, at 345671e.
>Â
> A side note:
> Recall that with the menu patch set tests, I found that the baseline
> reference performance for the pipe test on one core had changed
> significantly (worse - Kernel 4.19-rc1). Well, now it has changed
> significantly again (better, and even significantly better than it
> was for 4.18). 4.18 ~4.8 uSec/loop; 4.19 ~5.2 uSec/loop; 4.19+
> (345671e) 4.2 uSec/loop.
>Â
> This V2 is pretty good. All of the tests that I run gave similar
> performance and power use between the baseline reference and V2.
> I couldn't find any issues with the decay stuff, and I tried.
> (sorry, I didn't do pretty graphs.)
>Â
> After reading Giovanni's reply the other day, I tried the
> Phoronix dbench test: 12 clients resulted in similar performance,
> But TEOv2 used a little less processor package power; 256 clients
> had about -7% performance using TEOv2, but (my numbers are not
> exact) also used less processor package power.

Uhm, I see. The results I've got vary between machines; that could
depend on the CPU type. What is your machine processor model (or
microarchitecture, see the search box at the website https://ark.intel.com ),
and how many logical cores does it have?

For the record, in my previous email I wrote that my script runs dbench with
up to NUMCPUS*8 clients, but that's misleading; indeed for the 48-cores
machines I had runs with 1, 2, 4, 8, 16, 32 and 64 clients.
https://lore.kernel.org/lkml/1541010981.3423.2.camel@xxxxxxx/

The sequence is generated with

ÂÂÂÂCLIENT=1
ÂÂÂÂDBENCH_MAX_CLIENTS=$((NUMCPUS*8))

ÂÂÂÂwhile [ $CLIENT -le $DBENCH_MAX_CLIENTS ]; do

ÂÂÂÂÂÂÂÂÂÂÂÂ./bin/dbench [...] $CLIENT

ÂÂÂÂÂÂÂÂÂÂÂÂif [ $CLIENT -lt $NUMCPUS ]; then
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂCLIENT=$((CLIENT*2))
ÂÂÂÂÂÂÂÂÂÂÂÂelse
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂCLIENT=$((CLIENT*8))
ÂÂÂÂÂÂÂÂÂÂÂÂfi
ÂÂÂÂdone

In practice the max number of clients I get is slightly below NUMCPUS*2 to
reach saturation. I write this as I read you ran it with 256 clients but I
never went that high.

>Â
> On 2018.10.31 11:36 Giovanni Gherdovich wrote:
>Â
> > Something I'd like to do now is verify that "teo"'s predictions
> > are better than "menu"'s; I'll probably use systemtap to make
> > some histograms of idle times versus what idle state was chosen
> > -- that'd be enough to compare the two.
>Â
> I don't know what a "systemtap" is, but I have (crude) tools to
> post process trace data into histograms data. I did 5 minute
> traces during the 12 client Phoronix dbench test and plotted
> the results, [1]. Sometimes, to the right of the autoscaled
> graph is another with fixed scaling. Better grouping of idle
> durations with TEOv2 are clearly visible.
>Â
> ... Doug
>Â
> [1] http://fast.smythies.com/linux-pm/k419p/histo_compare.htm

Oh, that's interesting, thanks. Can you post the break-even residency times and
exit latencies for your CPUs? On my Skylake test machine I get this from sysfs:

$ cd /sys/devices/system/cpu/cpu0/cpuidle
$ for state in * ; do
echo -e \
"STATE: $state\t\
DESC: $(cat $state/desc)\t\
NAME: $(cat $state/name)\t\
LATENCY: $(cat $state/latency)\t\
RESIDENCY: $(cat $state/residency)"
done

STATE: state0ÂÂÂDESC: CPUIDLE CORE POLL IDLEÂÂÂÂNAME: POLLÂÂÂÂÂÂLATENCY: 0ÂÂÂÂÂÂRESIDENCY: 0
STATE: state1ÂÂÂDESC: MWAIT 0x00ÂÂÂÂÂÂÂÂNAME: C1ÂÂÂÂÂÂÂÂLATENCY: 2ÂÂÂÂÂÂRESIDENCY: 2
STATE: state2ÂÂÂDESC: MWAIT 0x01ÂÂÂÂÂÂÂÂNAME: C1EÂÂÂÂÂÂÂLATENCY: 10ÂÂÂÂÂRESIDENCY: 20
STATE: state3ÂÂÂDESC: MWAIT 0x10ÂÂÂÂÂÂÂÂNAME: C3ÂÂÂÂÂÂÂÂLATENCY: 70ÂÂÂÂÂRESIDENCY: 100
STATE: state4ÂÂÂDESC: MWAIT 0x20ÂÂÂÂÂÂÂÂNAME: C6ÂÂÂÂÂÂÂÂLATENCY: 85ÂÂÂÂÂRESIDENCY: 200
STATE: state5ÂÂÂDESC: MWAIT 0x33ÂÂÂÂÂÂÂÂNAME: C7sÂÂÂÂÂÂÂLATENCY: 124ÂÂÂÂRESIDENCY: 800
STATE: state6ÂÂÂDESC: MWAIT 0x40ÂÂÂÂÂÂÂÂNAME: C8ÂÂÂÂÂÂÂÂLATENCY: 200ÂÂÂÂRESIDENCY: 800

At the bottom of the email at
https://lore.kernel.org/lkml/4168371.zz0pVZtGOY@xxxxxxxxxxxxxx/
Rafael explains how the sysfs residencies are important to understand the
histograms.

Thanks,
Giovanni