sched/fair: which tasks should nr_pref_llc_running be compared against?

[Zhan Xusheng]

Reading alb_break_llc() I cannot tell which set of tasks the equality is
meant to cover:

	/* kernel/sched/fair.c:10756 */
	if (env->src_rq->nr_pref_llc_running &&
	    env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable) {

The two counters track different sets.  nr_pref_llc_running is maintained
from account_entity_enqueue() and account_entity_dequeue() at fair.c:4522
and 4538, beside cfs_rq->nr_queued++/-- at 4525 and 4541, so it follows
queued tasks.  h_nr_runnable leaves out delay-dequeued entities:
set_delayed() decrements it at 6398 while the entity stays queued and
clear_delayed() restores it at 6418, neither going through
account_entity_dequeue().

So with DELAY_DEQUEUE a task that has just gone to sleep holds
nr_pref_llc_running above h_nr_runnable until it is dequeued for real, the
equality cannot hold, and alb_break_llc() returns false, which stops
active load balance from honouring the LLC preference.  The counter is a
superset of the other, so the error is one-sided: the check can fail to
protect but never protects wrongly.

Comparing against cfs.h_nr_queued would make both sides agree and read as
"every queued fair task prefers this LLC".  Keeping runnable semantics
would instead mean maintaining nr_pref_llc_running from set_delayed() and
clear_delayed(), which may be the better fit, since only runnable tasks
are candidates for active load balance and a task on its way to sleep is
not one.  Which did you have in mind?

This is from reading the code rather than an observed failure, and I could
not exercise the path in a two-socket qemu guest either: with cache aware
scheduling on and one process of 6 busy and 10 sleeping threads,
p->preferred_llc was never assigned, so nr_pref_llc_running stayed 0
throughout while alb_break_llc() itself ran 21 times.  Nearly every
account_mm_sched() call returned before the mm->sc_stat check, which makes
me suspect update_se() bails on delta_exec <= 0 under emulation before the
accounting is reached.  If a guest cannot exercise this, that is worth
knowing on its own, since it is also how a mismatch like this stays
hidden.

Thanks,
Zhan Xusheng