Re: [PATCH 2/4] sched/fair: Introduce idle enter/exit balance callbacks

From: Jiri Olsa
Date: Mon Jun 20 2016 - 12:37:43 EST


On Mon, Jun 20, 2016 at 04:30:03PM +0200, Peter Zijlstra wrote:
> On Mon, Jun 20, 2016 at 02:15:12PM +0200, Jiri Olsa wrote:
> > Introducing idle enter/exit balance callbacks to keep
> > balance.idle_cpus_mask cpumask of current idle cpus
> > in system.
> >
> > It's used only when REBALANCE_AFFINITY feature is
> > switched on. The code functionality of this feature
> > is introduced in following patch.
> >
> > Signed-off-by: Jiri Olsa <jolsa@xxxxxxxxxx>
> > ---
> > kernel/sched/fair.c | 32 ++++++++++++++++++++++++++++++++
> > kernel/sched/idle.c | 2 ++
> > kernel/sched/sched.h | 3 +++
> > 3 files changed, 37 insertions(+)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index f19c9435c64d..78c4127f2f3a 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -7802,6 +7802,37 @@ static inline int on_null_domain(struct rq *rq)
> > return unlikely(!rcu_dereference_sched(rq->sd));
> > }
> >
> > +static struct {
> > + cpumask_var_t idle_cpus_mask;
> > + atomic_t nr_cpus;
> > +} balance ____cacheline_aligned;
>
> How is this different from the nohz idle cpu mask?

well, the nohz idle cpu mask is deep in the nohz code:

tick_irq_exit
if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
if (!in_interrupt())
tick_nohz_irq_exit
if (ts->inidle) {
__tick_nohz_idle_enter(ts)
tick_nohz_idle_enter
__tick_nohz_idle_enter
if (can_stop_idle_tick(cpu, ts)) {
tick_nohz_stop_sched_tick
if (ts->tick_stopped)
nohz_balance_enter_idle(cpu) {
set idle mask for cpu
}
} else {
tick_nohz_full_update_tick(ts);
if (can_stop_full_tick(ts))
tick_nohz_stop_sched_tick
if (ts->tick_stopped)
nohz_balance_enter_idle(cpu) {
set idle mask for cpu
}
}

cpu_idle_loop
tick_nohz_idle_enter
__tick_nohz_idle_enter(ts)
tick_nohz_idle_enter
__tick_nohz_idle_enter
if (can_stop_idle_tick(cpu, ts)) {
tick_nohz_stop_sched_tick
if (ts->tick_stopped)
nohz_balance_enter_idle(cpu) {
set idle mask for cpu
}


sched_cpu_dying
nohz_balance_exit_idle(cpu) {
unset idle mask for cpu
}

trigger_load_balance
nohz_kick_needed
nohz_balance_exit_idle(cpu) {
unset idle mask for cpu
}

... I might have missed some of the paths


so it might not be that easy to switch it to use the easy
change I added as part of this RFC, but AFAIU it should be
the same idle mask, but this approach might be too naive
and miss some idle enter/exit paths.. CC-ing Frederic

thanks,
jirka