Re: [RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Tobias Schaffner]

Hi Gabriele,

thanks for the quick reply and the motivating words.

On 8/27/26 10:18, Gabriele Monaco wrote:
> On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
> > Hi Steven, Gabriele,
> >
> > I am using the RV system for a downstream project called EVL (aka Xenomai4).
> > It is a small co-kernel system, small enough that I can model the whole
> > co-kernel with RV instead of just a fragment of it.
>
> Hi Tobias,
>
> Haven't heard about Xenomai in a while.. That sounds very interesting.
>
> Since it's something out-of-tree you likely don't want to submit monitors
> upstream, but if you can trigger BPF events from your co-kernel you may want to
> have a look at RV BPF monitors, which I'm likely going to submit in a couple of
> days.

Exactly. The monitors themselves will live downstream in the Dovetail 
and EVL projects.

I will definitely have a look at the RV BPF monitors!

> Otherwise we can always get out-of-tree monitors as modules, currently not
> supported just out of a missing use-case.
>
> > The patches in this series add a per-monitor "stats" file that exposes how
> > often an edge has been visited and the dwell time in the prior state for
> > per-cpu monitors.
>
> The idea crossed my mind a few times, good you came up with an implementation!
>
> > This allows me to not only check if the model is correct but also to see if
> > all paths are exercised by my stress tooling and to decompose the latency of
> > the co-kernel's wakeup path.
>
> Your first need (see if all paths are exercised) doesn't really require the stat
> to be per-monitor-instance (i.e. per-cpu) right?
> You could have a global accumulator matrix, at least for count and sum where you
> can be atomic. This could extend the feature to other types of monitors.

Yes that's correct.

For the dwell timings, though, keeping it per-cpu matters to make
isolation effects visible. I use sum and count there to derive
the average.

> Just brainstorming here though.
>
> > I think that some of the upstream monitors like e.g. the sts monitor could
> > also
> > profit from this.
> >
> > As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
> > kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
> > Only the edges the automaton actually takes are non-zero:
> >
> >    # cat monitors/sched/sts/stats
> >    # cpu edge label count max_ns sum_ns
> >    0 0 can_sched:irq_disable 367732 3998500 14773091250
> >    0 4 can_sched:schedule_entry 40282 398750 45510750
> >    0 7 cant_sched:irq_enable 367732 32500 532791750
> >    0 8 cant_sched:irq_entry 26188 6250 38357250
> >    0 13 disable_to_switch:irq_enable 4933 12500 12275000
> >    0 14 disable_to_switch:irq_entry 31 2250 35250
> >    0 15 disable_to_switch:sched_switch 35373 26750 111615750
> >    0 18 enable_to_exit:irq_disable 373 3250 649750
> >    0 19 enable_to_exit:irq_enable 373 20000 1676250
> >    0 20 enable_to_exit:irq_entry 309 4750 420000
> >    0 23 enable_to_exit:schedule_exit 40298 3000 37335000
> >    0 25 in_irq:irq_enable 31 12250 140750
> >    0 30 scheduling:irq_disable 40337 18000 45449500
> >    0 37 switching:irq_enable 35373 9250 72471000
> >    [ cpus 1-2 omitted ]
> >    3 0 can_sched:irq_disable 567 3983750 300860250
> >    3 4 can_sched:schedule_entry 150 1500 171000
> >    3 7 cant_sched:irq_enable 567 21500 1777500
> >    3 8 cant_sched:irq_entry 76 6500 164750
> >    3 15 disable_to_switch:sched_switch 150 4250 413750
> >    3 23 enable_to_exit:schedule_exit 150 1500 147750
> >    3 30 scheduling:irq_disable 150 1500 166000
> >    3 37 switching:irq_enable 150 3750 277000
> >
> > The sts model splits the schedule->switch path into separate states, so the
> > per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
> > prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
> > and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
> > counts also line up with the model, e.g. every irq_disable in can_sched has a
> > matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
> > take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
> > is tighter still (~4us), so the same file also makes the effect of cpu
> > isolation visible per cpu.
> >
> > The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
> > enabled, so existing setups are unaffected. When enabled, each accepted
> > transition adds one local_clock() and a few lock-free local64_t updates. With
> > the config off there is no code on the hot path at all.
> >
> > I focused on per-cpu monitors as a first step. Per-task and per-object
> > monitors would have to aggregate entities that share a cpu, which is harder
> > to get right, so I left them out for now.
>
> As said before, I haven't really played with an implementation but I think
> atomic types may help here.
>
> > What is your opinion on this? Do you think this is worth getting upstreamed?
>
> I will have a look at your patches but it's definitely something I'd want.

Looking forward to your input! I am very open to guidance on how to get
this right.

Best,
Tobias

> Thanks,
> Gabriele
>
> > Thanks for taking a look,
> > Tobias
> >
> > Tobias Schaffner (3):
> >    rv: add per-edge dwell-time statistics primitive
> >    rv: add per-monitor edge-stat facility and stats file
> >    rv: collect per-edge dwell time for per-cpu DA/HA monitors
> >
> >   .../trace/rv/runtime-verification.rst         |  24 ++++
> >   MAINTAINERS                                   |   1 +
> >   include/linux/rv.h                            |  18 +++
> >   include/linux/rv_edge_stat.h                  |  60 +++++++++
> >   include/rv/da_monitor.h                       |  53 ++++++++
> >   kernel/trace/rv/Kconfig                       |  11 ++
> >   kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
> >   tools/testing/selftests/verification/config   |   1 +
> >   .../verification/test.d/rv_edge_stats.tc      |  32 +++++
> >   9 files changed, 315 insertions(+), 1 deletion(-)
> >   create mode 100644 include/linux/rv_edge_stat.h
> >   create mode 100644
> > tools/testing/selftests/verification/test.d/rv_edge_stats.tc