Re: [PATCH 6/6] sched_ext/scx_qmap: Pick idle CPU for direct dispatch on !wakeup enqueues

[Tejun Heo]

Hello,

On Wed, Jul 10, 2024 at 02:25:23PM -0500, David Vernet wrote:
...
> > +	/* if !WAKEUP, select_cpu() wasn't called, try direct dispatch */
> > +	if (!(enq_flags & SCX_ENQ_WAKEUP) &&
> > +	    (cpu = pick_direct_dispatch_cpu(p, scx_bpf_task_cpu(p))) >= 0) {
> > +		__sync_fetch_and_add(&nr_ddsp_from_enq, 1);
> > +		scx_bpf_dispatch(p, SCX_DSQ_LOCAL_ON | cpu, slice_ns, enq_flags);
> > +		return;
> > +	}
> 
> Hmm, will this be a typical pattern for how this is used? I'd expect
> ops.select_cpu() and ops.enqueue() to quite often be nearly the same
> implementation. Meaning you would e.g. try to find an idle core in both, and do
> SCX_DSQ_LOCAL_ON, with the difference being that you'd just return the cpu and
> save the extra lock juggling if you did it on the ops.select_cpu() path. Not a
> huge deal given that it's just an example scheduler, but it might be a good
> idea to try and mirror typical use cases for that reason as well so readers get
> an idea of what a typical pattern would look like.

scx_qmap is a bit special in that it wants to be able to skip n'th tasks to
test the stall detection mechanism. We can't express that in select_cpu() as
not dispatching in select_cpu() just means that enqueue() will be invoked,
so it just funnels all tasks, even direct dispatch ones, to enqueue(), so it
looks a bit different from other schedulers. scx_qmap is mostly used to
exercise different code paths and is rather messy for anyone to use it as a
template. It'd probably be useful to note that in the comment.

Thanks.

-- 
tejun