Re: [PATCH v2 06/13] sched: Store maximum per-cpu capacity in root domain

[Dietmar Eggemann]

On 12/07/16 12:42, Peter Zijlstra wrote:
> On Mon, Jul 11, 2016 at 05:16:06PM +0100, Dietmar Eggemann wrote:
>> On 11/07/16 11:18, Peter Zijlstra wrote:
>>> On Wed, Jun 22, 2016 at 06:03:17PM +0100, Morten Rasmussen wrote:
>>>> @@ -6905,11 +6906,19 @@ static int build_sched_domains(const struct cpumask *cpu_map,
>>>>  	/* Attach the domains */
>>>>  	rcu_read_lock();
>>>>  	for_each_cpu(i, cpu_map) {
>>>> +		rq = cpu_rq(i);
>>>>  		sd = *per_cpu_ptr(d.sd, i);
>>>>  		cpu_attach_domain(sd, d.rd, i);
>>>> +
>>>> +		if (rq->cpu_capacity_orig > rq->rd->max_cpu_capacity)
>>>> +			rq->rd->max_cpu_capacity = rq->cpu_capacity_orig;
>>>>  	}
>>>
>>> Should you not set that _before_ cpu_attach_domain(), such that the
>>> state is up-to-date when its published?
>>
>> yes, much better.
>>
>>> Also, since its lockless, should we not use {READ,WRITE}_ONCE() with it?
>>
>> You mean for rq->rd->max_cpu_capacity ? IMHO, there is a data dependency
>> between the read and the write and the code only runs on one cpu.
>>
>> I assume here that this is related to item 2 'Overlapping loads and
>> stores within a particular CPU ...' in GUARANTEES of
>> doc/Documentation/memory-barriers.txt.
>>
>> Do I miss something?
> 
> Well, the value 'rd->max_cpu_capacity' is read by all CPUs attached to
> the root_domain, right? So CPUs already attached can observe this change
> when we update the value, we want them to observe either the old or the
> new max value, not a random mix of bytes.
> 
> {READ,WRITE}_ONCE() ensure whole word load/store, iow they avoid
> load/store-tearing.
> 

OK, thanks, will add them.

So this maps to the point '(*) For aligned memory locations whose size
allows them to be accessed with a single memory-reference instruction,
prevents "load tearing" and "store tearing," ...' under 'The READ_ONCE()
and WRITE_ONCE() functions can prevent any number of optimizations ...'
section in the 'COMPILER BARRIER' paragraph in
Documentation/memory-barriers.txt.