[PATCH] sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callers

[Zhan Xusheng]

From: Zhan Xusheng <zhanxusheng@xxxxxxxxxx>

pick_task_fair() and yield_task_fair() call update_curr(&rq->cfs) to bring
curr up to date before they look at the eevdf state.  With cgroups that
does not happen: update_curr() reads ->h_curr, which on the root cfs_rq is
the top level group entity, and returns at the !entity_is_task() check
before touching vruntime.  Both then read ->curr, so the guard and the
update disagree about which entity they mean.

Counting how often ->h_curr and ->curr differ at pick_task_fair(), on one
CPU for 10s with three busy tasks and one 200us-periodic task:

  all tasks in the root cgroup        43321 calls,     0 no-ops
  busy tasks in G0, periodic in G1    45211 calls, 45193 no-ops

Whether that matters depends on what precedes the pick.  Since
commit 68e37487810a ("sched/fair: Fix flat hierarchy") the tick and
enqueue/dequeue all update curr correctly, so on the normal reschedule
path only the microseconds between those and the pick are missing, and I
could not measure a latency difference there.  Three paths have nothing
before them on that rq though:

  - pick_task() on the sibling rqs of a core under core scheduling
    (kernel/sched/core.c), which updates that rq's clock first for
    exactly this reason
  - fair_server_pick_task()
  - yield_task_fair(), where the stale value feeds the entity_eligible()
    test that guards forfeiting the remaining vruntime

There curr can be a full tick behind, as it was before that commit.

No new behaviour for the entity being updated: without cgroups ->h_curr
is already the task, so these two call sites already run the full
update_curr() including update_deadline(), dl_server_update() and the
resched_curr_lazy() at the end.  This makes the cgroup case do the same.

Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")
Signed-off-by: Zhan Xusheng <zhanxusheng@xxxxxxxxxx>
---
 kernel/sched/fair.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 001140132a7d..3ea9bb5c3b47 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10052,7 +10052,7 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 
 	/* Might not have done put_prev_entity() */
 	if (cfs_rq->curr && cfs_rq->curr->on_rq)
-		update_curr(cfs_rq);
+		update_curr_eevdf(cfs_rq);
 
 	se = pick_next_entity(rq, true);
 	if (!se)
@@ -10155,7 +10155,7 @@ static void yield_task_fair(struct rq *rq)
 	/*
 	 * Update run-time statistics of the 'current'.
 	 */
-	update_curr(cfs_rq);
+	update_curr_eevdf(cfs_rq);
 	/*
 	 * Tell update_rq_clock() that we've just updated,
 	 * so we don't do microscopic update in schedule()
-- 
2.43.0