Re: Query regarding work scheduling

[Subbaraya Sundeep]

Hi Steven,

On 2025-08-14 at 17:45:50, Steven Rostedt (rostedt@xxxxxxxxxxx) wrote:
> On Thu, 14 Aug 2025 03:54:58 +0000
> Subbaraya Sundeep <sbhatta@xxxxxxxxxxx> wrote:
> 
> > > Difficult to tell where the latencies are coming from. Maybe you can use
> > > something like https://github.com/josefbacik/systing to look further into
> > > it? All the scheduling events are tracked by default and you should be able
> > > to add tracepoints and other events relatively easily. You can also set  
> 
> > Thanks for the reply. I am using simple busybox to avoid overhead of any other apps
> > or deamons running in background and taking CPU time in between.
> > I will try building systing and running it. 6.16 histogram shows that it
> > is not one high latency event causing overall latency but bunch of small
> > latencies are adding up and causing big latency.
> > I suspect this has something to do with EEVDF scheduling since this behavior is
> > seen from 6.6 (please note I may be wrong completly).
> > Are there any methods or options with which I can bring back CFS scheduling behavior
> > maybe with the knobs in /sys/kernel/debug/sched/features as a quick check? 
> 
> You could also use tracefs as that works on busybox:
> 
>  # echo 0 > /sys/kernel/tracing/tracing_on
>  # echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable
> [ and perhaps even more events ]
>  # echo 1 > /sys/kernel/tracing/tracing_on
>  # <run test>; echo 0 > /sys/kernel/tracing/tracing_on
>  # cat /sys/kernel/tracing/trace

Thank you for tracefs. I use it extensively and using timestamps in trace
data calculated the latencies.
> 
> You could even make it a trace.dat file:
> 
>  # mkdir /tmp/tracing
>  # cp -r /sys/kernel/tracing/events /tmp/tracing/
>  # cp -r /proc/kallsyms /tmp/tracing/
> [ have bs be PAGE_SIZE for your architecture ]
>  # dd bs=4096 if=/sys/kernel/tracing/per_cpu/cpu0/trace_pipe_raw of=/tmp/tracing/trace0.raw
>  # cd /tmp
>  # tar cvf trace.tar tracing
> 
> Copy trace.tar to a desktop and extract it.
> 
>  $ cd /tmp
>  $ tar xvf trace.tar
> [ Make sure you have the latest trace-cmd installed ]
>  $ trace-cmd restore -t /tmp/tracing/ -k /tmp/tracing/kallsyms -o /tmp/trace.dat /tmp/tracing/trace0.raw 
>  $ trace-cmp report /tmp/trace.dat
> 
> Now you can send us the trace.dat file and we could analyze it more.
> 
> You could also enable more events than just sched_switch, like sched_waking
> and such.
> 
I am on it. Thanks for the commands and your time.

Sundeep
> -- Steve