Re: [PATCH v3 3/3] sched_ext: Allow scx_bpf_reenqueue_local() to be called from anywhere

[Kuba Piecuch]

Hi Tejun,

I think with the proposed implementation, using scx_bpf_reenqueue_local()
from arbitrary contexts can have highly non-intuitive effects.

For example, consider ops.enqueue() for a hypothetical userspace scheduler:

void BPF_STRUCT_OPS(example_enqueue, struct task_struct *p, u64 enq_flags)
{
	if (p->pid == user_scheduler_pid()) {
		/*
		 * Remove existing tasks from the local DSQ so that
		 * the userspace scheduler can schedule different tasks
		 * before them.
		 */
		scx_bpf_reenqueue_local();
		/*
		 * Dispatch the user scheduler directly to the local DSQ.
		 */
		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
	}
	...
}

I'm not arguing this is the way it should be written, but AFAIK it's perfectly
legal.

Since we're doing a direct dispatch, the user scheduler task will be
inserted into the dispatch queue in enable_task_scx(), without dropping the rq
lock between example_enqueue() and the insertion, which means reenq_local()
will run afterwards (since it's deferred using irq_work), removing all tasks
from the DSQ, including the userspace scheduler.

A similar problem arises even if we don't do direct dispatch and drop the rq
lock after example_enqueue(): since dispatching and reenq_local() are deferred
using different irq_work entries, and irq_work_run() processes entries from
newest to oldest, dispatching will be handled before reenq_local(), yielding
the same result.

The user may be unaware of this behavior (it's not mentioned anywhere) and
expect the reenqueue to happen before dispatching the new task.

I think at the very least we should make users aware of this in the comment
for scx_bpf_reenqueue_local___v2().

Best,
Kuba