Re: [PATCH] x86: Run checksumming in parallel accross multiple alu's

[Doug Ledford]

On 10/19/2013 04:23 AM, Ingo Molnar wrote:
> 
> * Doug Ledford <dledford@xxxxxxxxxx> wrote:
>> All prefetch operations get sent to an access queue in the memory 
>> controller where they compete with both other reads and writes for the 
>> available memory bandwidth.  The optimal prefetch window is not a factor 
>> of memory bandwidth and latency, it's a factor of memory bandwidth, 
>> memory latency, current memory access queue depth at time prefetch is 
>> issued, and memory bank switch time * number of queued memory operations 
>> that will require a bank switch.  In other words, it's much more complex 
>> and also much more fluid than any static optimization can pull out. 
>> [...]
> 
> But this is generally true of _any_ static operation - CPUs are complex, 
> workloads are complex, other threads, CPUs, sockets, devices might 
> interact, etc.
> 
> Yet it does not make it invalid to optimize for the isolated, static 
> usecase that was offered, because 'dynamism' and parallelism in a real 
> system will rarely make that optimization completely invalid, it will 
> typically only diminish its fruits to a certain degree (for example by 
> causing prefetches to be discarded).

So, prefetches are a bit of a special beast in that, if they are done
incorrectly, they can actually make the overall system slower than if we
didn't do anything at all.  If you are talking about anything other than
prefetch I would agree with you.  With prefetch, as much as possible,
you need to mimic the environment for which you are optimizing.  Neil's
test kernel module just called csum_partial on a bunch of memory pages.
 The actual usage pattern of csum_partial though is that it will be used
while, most likely, there is ongoing DMA of data packets across the
network interface.  It is very unlikely that we could care less about
the case of optimizing csum_partial for no network activity since
csum_partial is always going to be the result of network activity and
unlikely to happen in isolation.  As such, my suggestion about a kernel
compile was to create activity across the PCI bus to hard drives,
mimicking network interface DMA traffic.  You could also run a netperf
instance instead.  I just don't agree with optimizing it without
simultaneous DMA traffic as that particular case if likely to be a
rarity, not the norm.

> What I was objecting to strongly here was to measure the _wrong_ thing, 
> i.e. the cache-hot case. The cache-cold case should be measured in a low 
> noise fashion, so that results are representative. It's closer to the real 
> usecase than any other microbenchmark. That will give us a usable speedup 
> figure and will tell us which technique helped how much and which 
> parameter should be how large.

Cold cache, yes.  Low noise, yes.  But you need DMA traffic at the same
time to be truly representative.

>> [...]  So every time I see someone run a series of micro- benchmarks 
>> like you just did, where the system was only doing the micro- benchmark 
>> and not a real workload, and we draw conclusions about optimal prefetch 
>> distances from that test, I cringe inside and I think I even die... just 
>> a little.
> 
> So the thing is, microbenchmarks can indeed be misleading - and as in this 
> case the cache-hot claims can be outright dangerously misleading.

So can non-DMA cases.

> But yet, if done correctly and interpreted correctly they tell us a little 
> bit of the truth and are often correlated to real performance.
> 
> Do microbenchmarks show us everything that a 'real' workload inhibits? Not 
> at all, they are way too simple for that. They are a shortcut, an 
> indicator, which is often helpful as long as not taken as 'the' 
> performance of the system.
> 
>> A better test for this, IMO, would be to start a local kernel compile 
>> with at least twice as many gcc instances allowed as you have CPUs, 
>> *then* run your benchmark kernel module and see what prefetch distance 
>> works well. [...]
> 
> I don't agree that this represents our optimization target. It may 
> represent _one_ optimization target. But many other important usecases 
> such as a dedicated file server, or a computation node that is 
> cache-optimized, would unlikely to show such high parallel memory pressure 
> as a GCC compilation.

But they will *all* show network DMA load, not quiescent DMA load.

>> [...]  This distance should be far enough out that it can withstand 
>> other memory pressure, yet not so far as to constantly be prefetching, 
>> tossing the result out of cache due to pressure, then fetching/stalling 
>> that same memory on load.  And it may not benchmark as well on a 
>> quiescent system running only the micro-benchmark, but it should end up 
>> performing better in actual real world usage.
> 
> The 'fully adversarial' case where all resources are maximally competed 
> for by all other cores is actually pretty rare in practice. I don't say it 
> does not happen or that it does not matter, but I do say there are many 
> other important usecases as well.
> 
> More importantly, the 'maximally adversarial' case is very hard to 
> generate, validate, and it's highly system dependent!

This I agree with 100%, which is why I tend to think we should scrap the
static prefetch optimizations entirely and have a boot up test that
allows us to find our optimum prefetch distance for our given hardware.

> Cache-cold (and cache hot) microbenchmarks on the other hand tend to be 
> more stable, because they typically reflect current physical (mostly 
> latency) limits of CPU and system technology, _not_ highly system 
> dependent resource sizing (mostly bandwidth) limits which are very hard to 
> optimize for in a generic fashion.
> 
> Cache-cold and cache-hot measurements are, in a way, important physical 
> 'eigenvalues' of a complex system. If they both show speedups then it's 
> likely that a more dynamic, contended for, mixed workload will show 
> speedups as well. And these 'eigenvalues' are statistically much more 
> stable across systems, and that's something we care for when we implement 
> various lowlevel assembly routines in arch/x86/ which cover many different 
> systems with different bandwidth characteristics.
> 
> I hope I managed to explain my views clearly enough on this.
> 
> Thanks,
> 
> 	Ingo
> 


-- 
Doug Ledford <dledford@xxxxxxxxxx>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Attachment: signature.asc
Description: OpenPGP digital signature