Re: [PATCH 0/10 v2] sched/fair: Fix statistics with delayed dequeue

[K Prateek Nayak]

Hello Vincent, Mike,

On 12/2/2024 2:47 PM, Vincent Guittot wrote:
> On Sun, 1 Dec 2024 at 14:30, Mike Galbraith <efault@xxxxxx> wrote:
> > Greetings,
> >
> > On Fri, 2024-11-29 at 17:17 +0100, Vincent Guittot wrote:
> > > Delayed dequeued feature keeps a sleeping sched_entitiy enqueued until its
> > > lag has elapsed. As a result, it stays also visible in the statistics that
> > > are used to balance the system and in particular the field h_nr_running.
> > >
> > > This serie fixes those metrics by creating a new h_nr_queued that tracks
> > > all queued tasks. It renames h_nr_running into h_nr_runnable and restores
> > > the behavior of h_nr_running i.e. tracking the number of fair tasks that
> > >   want to run.
> > >
> > > h_nr_runnable is used in several places to make decision on load balance:
> > >    - PELT runnable_avg
> > >    - deciding if a group is overloaded or has spare capacity
> > >    - numa stats
> > >    - reduced capacity management
> > >    - load balance between groups
> >
> > I took the series for a spin in tip v6.12-10334-gb1b238fba309, but
> > runnable seems to have an off-by-one issue, causing it to wander ever
> > further south.
> >
> > patches 1-3 applied.
> >    .h_nr_runnable                 : -3046
> >    .runnable_avg                  : 450189777126
>
> Yeah, I messed up something around finish_delayed_dequeue_entity().
> I'm' going to prepare a v3

I was looking into this and I have the below diff so far that seems to
solve the post boot negative values of h_nr_runnable on my setup; it is
only lightly tested so far:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 87552870958c..423981e65aba 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5464,6 +5464,10 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
 static void set_delayed(struct sched_entity *se)
 {
 	se->sched_delayed = 1;
+
+	if (!entity_is_task(se))
+		return;
+
 	for_each_sched_entity(se) {
 		struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
@@ -5476,6 +5480,10 @@ static void set_delayed(struct sched_entity *se)
 static void clear_delayed(struct sched_entity *se)
 {
 	se->sched_delayed = 0;
+
+	if (!entity_is_task(se))
+		return;
+
 	for_each_sched_entity(se) {
 		struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
@@ -6977,7 +6985,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	struct cfs_rq *cfs_rq;
 	struct sched_entity *se = &p->se;
 	int h_nr_idle = task_has_idle_policy(p);
-	int h_nr_runnable = 0;
+	int h_nr_runnable = 1;
 	int task_new = !(flags & ENQUEUE_WAKEUP);
 	int rq_h_nr_queued = rq->cfs.h_nr_queued;
 	u64 slice = 0;
@@ -7124,8 +7132,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
 		p = task_of(se);
 		h_nr_queued = 1;
 		h_nr_idle = task_has_idle_policy(p);
-		if (!task_sleep && !task_delayed)
-			h_nr_runnable = !se->sched_delayed;
+		h_nr_runnable = !se->sched_delayed;
 	} else {
 		cfs_rq = group_cfs_rq(se);
 		slice = cfs_rq_min_slice(cfs_rq);
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index ab911d1335ba..f4ef5aaa4674 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -457,6 +457,7 @@ static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct t
 
 static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first)
 {
+	SCHED_WARN_ON(rq->cfs.h_nr_runnable);
 	update_idle_core(rq);
 	scx_update_idle(rq, true);
 	schedstat_inc(rq->sched_goidle);
--

I'm not sure if the change in dequeue_entities() is completely necessary
but I added it after seeing the (DEQUEUE_SLEEP | DEQUEUE_DELAYED) in
throttle_cfs_rq() but there the entity cannot possibly be a task so
perhaps that part is unnecessary ¯\_(ツ)_/¯

Still testing! Will keep an eye out for v3.

> > full set applied.
> >    .h_nr_runnable                 : -5707
> >    .runnable_avg                  : 4391793519526
> >
> >          -Mike

--
Thanks and Regards,
Prateek