[DISCUSSION] SCX_OPS_ALWAYS_ENQ_IMMED semantics clarification

[zhidao su]

Hi Tejun and all,

I'm working on understanding the sched_ext enqueue flow and noticed a
potential documentation gap regarding SCX_OPS_ALWAYS_ENQ_IMMED that I'd
like to clarify.

**Background:**

The flag comment in ext_internal.h L186-188 states:
  "If set, %SCX_ENQ_IMMED is assumed to be set on all local DSQ enqueues."

However, reviewing the implementation:
- The SCX_ENQ_IMMED injection happens only in scx_vet_enq_flags() L7812, 
  which is called exclusively from BPF kfunc paths 
  (scx_bpf_dsq_insert/scx_bpf_dsq_move_to_local).
- Kernel-side enqueues via do_enqueue_task() that take the `goto local` 
  path (L1883-1893) bypass scx_vet_enq_flags() entirely.

This means exiting and migration-disabled tasks using the kernel-side 
goto local path don't get SCX_ENQ_IMMED injected, and consequently 
don't get SCX_TASK_IMMED set in dsq_inc_nr().

The code comment at L1859-1862 acknowledges this:
  "exiting and migration-disabled tasks that skip ops.enqueue() below 
  will lose IMMED protection unless SCX_OPS_ENQ_EXITING / 
  SCX_OPS_ENQ_MIGRATION_DISABLED are set."

**Questions:**

1. Is this behavior intentional? Should "all local DSQ enqueues" in the 
   flag documentation be understood as "all BPF-path local DSQ enqueues"?

2. Should the flag documentation be updated to clarify this exception?

3. Or is the current behavior a gap that should be fixed by applying 
   ALWAYS_ENQ_IMMED semantics uniformly to kernel-side goto local paths 
   as well?

This would help clarify whether schedulers setting this flag can assume 
truly all local enqueues get IMMED protection, or only those through 
the BPF interface.

Thanks,
zhidao su