Re: [PATCH v2 06/13] sched: Store maximum per-cpu capacity in root domain

[Peter Zijlstra]

On Mon, Jul 11, 2016 at 05:16:06PM +0100, Dietmar Eggemann wrote:
> On 11/07/16 11:18, Peter Zijlstra wrote:
> > On Wed, Jun 22, 2016 at 06:03:17PM +0100, Morten Rasmussen wrote:
> >> @@ -6905,11 +6906,19 @@ static int build_sched_domains(const struct cpumask *cpu_map,
> >>  	/* Attach the domains */
> >>  	rcu_read_lock();
> >>  	for_each_cpu(i, cpu_map) {
> >> +		rq = cpu_rq(i);
> >>  		sd = *per_cpu_ptr(d.sd, i);
> >>  		cpu_attach_domain(sd, d.rd, i);
> >> +
> >> +		if (rq->cpu_capacity_orig > rq->rd->max_cpu_capacity)
> >> +			rq->rd->max_cpu_capacity = rq->cpu_capacity_orig;
> >>  	}
> > 
> > Should you not set that _before_ cpu_attach_domain(), such that the
> > state is up-to-date when its published?
> 
> yes, much better.
> 
> > Also, since its lockless, should we not use {READ,WRITE}_ONCE() with it?
> 
> You mean for rq->rd->max_cpu_capacity ? IMHO, there is a data dependency
> between the read and the write and the code only runs on one cpu.
> 
> I assume here that this is related to item 2 'Overlapping loads and
> stores within a particular CPU ...' in GUARANTEES of
> doc/Documentation/memory-barriers.txt.
> 
> Do I miss something?

Well, the value 'rd->max_cpu_capacity' is read by all CPUs attached to
the root_domain, right? So CPUs already attached can observe this change
when we update the value, we want them to observe either the old or the
new max value, not a random mix of bytes.

{READ,WRITE}_ONCE() ensure whole word load/store, iow they avoid
load/store-tearing.