Re: sched: tweak select_idle_sibling to look for idle threads

[Mike Galbraith]

On Mon, 2016-05-02 at 16:58 +0200, Peter Zijlstra wrote:
> On Mon, May 02, 2016 at 04:50:04PM +0200, Mike Galbraith wrote:
> > Oh btw, did you know single socket boxen have no sd_busy?  That
> > doesn't
> > look right.
> 
> I suspected; didn't bother looking at yet. The 'problem' is that the
> LLC
> domain is the top-most, so it doesn't have a parent domain. I'm sure
> we
> can come up with something if we can get this all working right.
> 
> And yes, I can get gains on various workloads with various options, I
> can even break all workloads, but I've so far completely failed on
> getting a win for everyone :/
> 
> In particular low count sysbench-psql (oltp test) vs tbench
> client==nr_cores is having me flummoxed for a bit.

When I was doing a lot of that, there was always considerable overlap
to reclaim with db stuff, and often other players.  With loalhost
tbench only a wee bit of overlap and two players, it's pretty close to
really being the synchronous load it advertises.  For tbench (and ilk),
the best thing you can do is completely kill select_idle_sibling() once
past ramp.  Hell, if you watch TCP_RR, pipe-test, whatever with much
less overlap, you'll ask yourself why the hell select_idle_sibling()
even exists for anything other than the original shared L2 case it was
born to serve. 

Unfortunately, the real world spoiler is that a sync wakeup means jack
diddly spit, a waker may very well suddenly be awakened on it's way to
sleepy land.. so much for sync wakeup, oh crap, we need to scale.  The
problem ain't simple, but the scheduler has to be to perform.  Catch22.

	-Mike