[PATCH] sched: Fix incorrect sched_stat_wait statistics for rt and dl

[luoliang]

From: Liang Luo <luoliang@xxxxxxxxxx>

The update_stats_wait_start_*() wrappers only record wait_start while
schedstat_enabled(). With schedstats disabled by default, a task
enqueued before they are enabled at runtime has a zero wait_start, and
its next __update_stats_wait_end() computes the wait time as

	rq_clock(rq) - 0

which is the time since boot. The bogus delta is folded into wait_max
and wait_sum and passed to trace_sched_stat_wait(). Neither wait_max
nor wait_sum ever shrinks, so a single bogus sample corrupts the
task's wait statistics until it exits.

Reproduced on mainline with a SCHED_RR task:

$ grep wait_max /proc/<pid>/sched
wait_max                                     :       1058533.555050

on a machine with ~1058s of uptime.

The fair class has been immune since commit b9c88f752268
("sched/fair: Improve the accuracy of sched_stat_wait statistics"),
which skips the accounting on a zero wait_start, but the check lives
in the fair wrapper. The rt and dl wrappers call
__update_stats_wait_end() directly and never picked it up.

Move the check into __update_stats_wait_end() so that every class and
call site shares it, and drop the now redundant one from the fair
wrapper.

Fixes: 57a5c2dafca8 ("sched/rt: Support schedstats for RT sched class")
Fixes: b5eb4a5f6521 ("sched/dl: Support schedstats for deadline sched class")
Signed-off-by: Liang Luo <luoliang@xxxxxxxxxx>
---
An equivalent fix was proposed by Zhang Qiao in 2024 but never merged:

https://lore.kernel.org/r/20240322081521.2687856-1-zhangqiao22@xxxxxxxxxx

 kernel/sched/fair.c  |  9 ---------
 kernel/sched/stats.c | 12 +++++++++++-
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 7455a83a6a99..bf638eb359c4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -2116,15 +2116,6 @@ update_stats_wait_end_fair(struct cfs_rq *cfs_rq, struct sched_entity *se)
 
 	stats = __schedstats_from_se(se);
 
-	/*
-	 * When the sched_schedstat changes from 0 to 1, some sched se
-	 * maybe already in the runqueue, the se->statistics.wait_start
-	 * will be 0.So it will let the delta wrong. We need to avoid this
-	 * scenario.
-	 */
-	if (unlikely(!schedstat_val(stats->wait_start)))
-		return;
-
 	if (entity_is_task(se))
 		p = task_of(se);
 
diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c
index d1c9429a4ac5..dce034b88c32 100644
--- a/kernel/sched/stats.c
+++ b/kernel/sched/stats.c
@@ -21,7 +21,17 @@ void __update_stats_wait_start(struct rq *rq, struct task_struct *p,
 void __update_stats_wait_end(struct rq *rq, struct task_struct *p,
 			     struct sched_statistics *stats)
 {
-	u64 delta = rq_clock(rq) - schedstat_val(stats->wait_start);
+	u64 delta;
+
+	/*
+	 * Tasks enqueued while schedstats were disabled have a zero
+	 * wait_start: the wait was never recorded. Skip it, as the delta
+	 * against 0 would be the time since boot.
+	 */
+	if (unlikely(!schedstat_val(stats->wait_start)))
+		return;
+
+	delta = rq_clock(rq) - schedstat_val(stats->wait_start);
 
 	if (p) {
 		if (task_on_rq_migrating(p)) {
-- 
2.43.0