Re: [PATCH 1/2] sched/doc: add a preemption model overview

[Sebastian Andrzej Siewior]

On 2026-09-20 10:20:02 [+0800], Quchaosheng wrote:
> The scheduler documentation describes the individual scheduling classes
> and various tuning knobs, but there is nothing that describes the
> preemption models themselves.  The only place where they are documented
> is the kernel-parameters entry for "preempt=", which explains the boot
> time parameter but not the models it selects.
> 
> Add Documentation/scheduler/sched-preemption.rst, covering the four
> models, which of them can be selected at runtime, and the mechanism
> behind PREEMPT_LAZY.  The lazy model in particular is easy to
> misunderstand: a lazy reschedule does not send a cross-CPU reschedule
> IPI, so it is only committed on the return to user space or on the next
> tick.  That makes the tick an upper bound on lazy preemption latency,
> and it means the usual real-time latency tools, which wake a pinned
> task on its own CPU, do not exercise it at all.
> 
> Signed-off-by: Quchaosheng <quchaosheng000406@xxxxxxx>
> ---
>  Documentation/scheduler/index.rst            |   1 +
>  Documentation/scheduler/sched-preemption.rst | 120 +++++++++++++++++++
>  2 files changed, 121 insertions(+)
>  create mode 100644 Documentation/scheduler/sched-preemption.rst
> 
> diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/index.rst
> index 17ce8d76befc..d6d75421756a 100644
> --- a/Documentation/scheduler/index.rst
> +++ b/Documentation/scheduler/index.rst
> @@ -23,5 +23,6 @@ Scheduler
>      sched-stats
>      sched-ext
>      sched-debug
> +    sched-preemption
>  
>      text_files
> diff --git a/Documentation/scheduler/sched-preemption.rst b/Documentation/scheduler/sched-preemption.rst
> new file mode 100644
> index 000000000000..fc6fd89789cf
> --- /dev/null
> +++ b/Documentation/scheduler/sched-preemption.rst
> @@ -0,0 +1,120 @@
> +=====================
> +Scheduler preemption
> +=====================
> +
> +The kernel can be built to run kernel code either uninterruptibly, or with
> +varying degrees of preemptibility.  These are the *preemption models*.

  These preemption models apply only to the kernel - a user-space task is
  always preempted by the scheduler regardless of the selected model.

> +
> +When CONFIG_PREEMPT_DYNAMIC is enabled the preemption model can additionally
> +be selected at boot time with the ``preempt=`` command line parameter, without
> +rebuilding the kernel.  See
> +Documentation/admin-guide/kernel-parameters.txt for the parameter itself.

  Additionally the preemption model can be inspected and changed via the
  /sys/kernel/debug/sched/preempt file (which is available for debugging
  purposes and may change).

> +
> +The models selectable at runtime are:
> +
> +	=========	====================================================
> +	none		No preemption of kernel code other than at explicit
> +			``cond_resched()`` / blocking points.
> +	voluntary	As ``none``, plus ``might_sleep()`` sites.
> +	full		Any section that is not explicitly preempt disabled
> +			may be preempted at any time.  Tasks also yield
> +			contended spinlocks.

What does "Tasks also yield contended spinlocks" mean?

> +	lazy		As ``full``, except that a reschedule requested by
> +			the fair scheduler does not interrupt the target
> +			CPU.  It is committed at the next return to user
> +			space or at the next tick, whichever comes first.

  Same as ``full` except that the scheduling request is delayed until the
  return to user space or the next tick, whichever comes first. This delay
  does not apply to real-time tasks.


> +	=========	====================================================
> +
> +Not every model is available on every kernel.  On a kernel that selects
> +ARCH_HAS_PREEMPT_LAZY, only ``full`` and ``lazy`` are offered and ``none`` and
> +``voluntary`` are rejected.  A PREEMPT_RT kernel likewise does not offer
> +``none`` or ``voluntary``.

I would rephrase it to
  On an architecture, that provides support lazy-preempt, only …

and maybe 
  A PREEMPT_RT kernel offers always ``full`` and ``lazy`` if provided by
  the architecture.

> +The model that is actually active is reported in the boot log::
> +
> +	Dynamic Preempt: full
> +
> +PREEMPT_LAZY
> +============
> +
> +``lazy`` is the interesting one, because its behaviour cannot be understood
> +from the "can this be preempted" question alone.
> +
> +When a wakeup makes a running CFS task preemptible, the fair scheduler calls
> +``resched_curr_lazy()``.  With the lazy model inactive that is just
> +``resched_curr()``; with ``lazy`` active it sets ``TIF_NEED_RESCHED_LAZY``
> +instead of ``TIF_NEED_RESCHED``.  The consequence is in ``__resched_curr()``:
> +the cross-CPU reschedule IPI is only sent for ``TIF_NEED_RESCHED``.
> +

This is a technical description of the code. From a bird's eye view:

  The scheduler becomes a wake up request for a task and marks it
  immediately as eligible for running. If there is an idle CPU which is
  suitable then it will move the task there. Otherwise the scheduler must
  decide which task to preempt. Once a decision has been made, the task,
  that needs to be removed from the CPU, gets a "NEED RESCHED" flag set.
  In tracing this is visible in the "need-resched" column and is denoted
  by a ``N`` for "NEED_RESCHED" or ``l`` for "NEED_RESCHED_LAZY" request.
  The next step for kernel is to honour this scheduling request. 
  A task executing in userland can always be preempted. A task executing
  in kernel can only be preempted when it is safe to do so.
  
  - No Forced Preemption
    Code paths with loops have often a "cond_resched()" function which
    performs scheduling.
  
  - Voluntary Kernel Preemption
    Same as in the previous model and additionally functions, which are
    known that they may block and schedule, have also such a
    "cond_resched()".
  
  - Preemptible Kernel
    The previously mention "cond_resched()" mechanics is not involved.
    Instead the kernel tracks preemptiblity and schedules after a
    scheduling request (NEED_RESCHED) once possible.
  
  - Scheduler controlled preemption model
    Same as in the previous model except that the default scheduling
    request is "NEED_RESCHED_LAZY". This scheduling request is not
    honoured immediately, even if possible, but delayed until the task is
    returning back to userland (allowing running to completion).
    This "NEED_RESCHED_LAZY" scheduling request will be turned into a
    "NEED_RESCHED" request on the next HZ tick should the task not
    schedule on its own before that. This is denoted in the tracing output
    by a "B".
    Real-time tasks use always the "NEED_RESCHED" mechanism to avoid any
    delays.

Wouldn't this do the job?

> +A lazy reschedule therefore does *not* interrupt the target CPU.  It is
> +committed later, in one of two ways:
> +
> +1. On the next return to user mode.  ``__exit_to_user_mode_loop()`` treats
> +   ``_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY`` as a reschedule request, and
> +   ``xfer_to_guest_mode_work()`` does the same for a vCPU returning to guest
> +   mode.
> +
> +2. In ``scheduler_tick()``, which promotes a pending
> +   ``TIF_NEED_RESCHED_LAZY`` to a full ``TIF_NEED_RESCHED`` once per tick.
> +
> +The tick is thus an upper bound on lazy preemption latency for CFS tasks: a
> +runnable CFS task that has been passed over is picked up within one tick, not
> +immediately.  The tick is guaranteed to run in the case that matters: while
> +more than one CFS task is runnable on the CPU, ``sched_can_stop_tick()``
> +refuses to stop the tick.  A lazy reschedule aimed at the idle task is promoted
> +to a full one instead.

This is an implementation detail I think.

> +That is the intended trade-off -- it avoids sending an IPI to every CPU that
> +has a runnable CFS task -- and it is why ``lazy`` gives up latency in exchange
> +for fewer inter-processor interrupts.

Maybe I miss the point but a wakeup never sends an IPI to every CPU. It
only sends one to the CPU which should run task in a cross-CPU wake-up.

> +The idle task is never delayed: ``__resched_curr()`` promotes a lazy
> +reschedule to a full one when the target is the idle task.

You never explained what an idle task and this is not visible in ps
either. Also an implementation detail.

> +Only the fair scheduler issues lazy reschedules.  RT and deadline tasks are
> +still preempted immediately.  ``lazy`` is therefore not a replacement for
> +PREEMPT_RT; it is a way to keep most of the responsiveness of ``full`` while
> +removing a large part of its IPI traffic.

Not sure it is the IPI traffic. Of course the IPI is absent on cross-CPU
wake-ups. The main point however is to allow "run-to-completion" for the
task. By pushing a scheduling request to the point where it returns to
userland, the "in-kernel-work" has been done. That means all locks have
been dropped so another task will not ask for a mutex which is held by
preempted task. Also it will not write-back cache hot memory which needs
to be dragged back once it schedules back-in.

> +
> +Debugging
> +=========
> +
> +The scheduler debugfs directory provides ``/sys/kernel/debug/sched/preempt``,
> +which lists the models that may be switched to; the active one is enclosed in
> +parentheses::
> +
> +	# cat /sys/kernel/debug/sched/preempt
> +	full (lazy)
> +
> +Writing a model name to the file switches to it.  A name that the kernel
> +cannot select (either because the model is not built in, or because it is not
> +available on this kernel) is rejected with EINVAL.  The ``preempt=``
> +boot parameter rejects such a value as well, but only prints an ``unsupported
> +mode`` warning and keeps the default model.

I added a small point upstairs, you have a bit more here, not sure what
to do about here. Reading and writing is sort of obvious.

> +Measuring
> +=========
> +
> +Two things routinely trip people up when measuring preemption latency.
> +
> +First, the classic real-time latency tools do not exercise lazy preemption at
> +all.  Lazy only applies to the fair class, so waking a SCHED_FIFO task takes
> +``resched_curr()`` and preempts immediately, whichever model is active.  And

The scheduling request happens immediately for SCHED_FIFO, yes. But
preemption does not happen immediately on NONE for instance.

> +when the woken task runs on the CPU the wakeup happens on, even a fair lazy
> +reschedule is committed as soon as that CPU returns to user mode, which for an
> +already runnable task is immediate.  For both reasons such workloads measure
> +the same under ``lazy`` and ``full``.

I don't comprehend that one.

> +
> +The difference only shows up when the woken task is *not* running on the CPU
> +where the wakeup happens.  In that case the waker requests the reschedule, and
> +under ``lazy`` that request is not turned into an IPI; the target CPU only
> +notices at its next tick.  A measurement that cannot distinguish ``none`` from
> +``full`` is usually not measuring the preemption model at all: in the absence
> +of ``lazy``, the reschedule IPI is sent regardless of the model and provides a
> +preemption point even on PREEMPT_NONE.

This is a detail that could be mentioned. However, despite a wakeup (as
of try_to_wake_up()) the scheduler can decide to keep the current task
on CPU (because it did not consume enough of its time slice) and not
schedule the runnable SCHED_OTHER task immediately. There is no
requirement that a freshly woke up task must be scheduled right away.

> +
> +Second, when driving cross-CPU wakeups from a periodic source, keep the wakeup
> +period away from integer multiples of the tick.  A period that is phase-locked
> +to the tick will hide (or invent) exactly the effect that is being measured.
> +Add jitter to the wakeup period, and confirm the active model from the
> +``Dynamic Preempt:`` line rather than assuming that ``preempt=`` took effect.

If you do timer-based wake ups for SCHED_OTHER tasks you also have the
"slack" value. Also you can have jitter from preemption or interrupts
disabled regions.

Sebastian