Re: [PATCH] sched: make update_sd_pick_busiest return true on a busier sd

[Vincent Guittot]

On 25 July 2014 16:02, Rik van Riel <riel@xxxxxxxxxx> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/23/2014 03:41 AM, Vincent Guittot wrote:
>
>> Regarding your issue with "perf bench numa mem" that is not spread
>> on all nodes, SD_PREFER_SIBLING flag (of DIE level) should do the
>> job by reducing the capacity of  "not local DIE" group at NUMA
>> level to 1 task during the load balance computation. So you should
>> have 1 task per sched_group at NUMA level.
>
> Looking at the code some more, it is clear why this does not
> happen. If sd->flags & SD_NUMA, then SD_PREFER_SIBLING will
> never be set.

I don't have a lot of experience on NUMA system and how their
sched_domain topology is described but IIUC, you don't have other
sched_domain level than NUMA ones ? otherwise the flag should be
present in one of the non NUMA level (SMT, MC or DIE)

>
>
> On a related note, that part of the load balancing code probably
> needs to be rewritten to deal with unequal group_capacity_factors
> anyway.
>
> Say that one group has a group_capacity_factor twice that of
> another group.
>
> The group with the smaller group_capacity_factor is overloaded
> by a factor 1.3. The larger group is loaded by a factor 0.8.
> This means the larger group has a higher load than the first
> group, and the current code in update_sd_pick_busiest will
> not select the overloaded group as the busiest one, due to not
> scaling load with the capacity...
>

AFAICT, sgs->avg_load is weighted by the capacity in update_sg_lb_stats

> static bool update_sd_pick_busiest(struct lb_env *env,
>                                    struct sd_lb_stats *sds,
>                                    struct sched_group *sg,
>                                    struct sg_lb_stats *sgs)
> {
>         if (sgs->avg_load <= sds->busiest_stat.avg_load)
>                 return false;
>
> I believe we may need to factor the group_capacity_factor
> into this calculation, in order to properly identify which
> group is busiest.
>
> However, if we do that we may need to get rid of the
> SD_PREFER_SIBLING hack that forces group_capacity_factor
> to 1 on domains that have SD_PREFER_SIBLING set.

I'm working on a patchset that get ride of capacity_factor (as
mentioned by Peter) and directly uses capacity instead. I should send
the v4 next week.

Vincent
>
> I suspect that should be ok though, if we make sure
> update_sd_pick_busiest does the right thing...
>
> - --
> All rights reversed
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJT0mOCAAoJEM553pKExN6DHq4H/2THfH33d+JYvfOq95OpGLaD
> HATAp8Dv0kTiGjnbZrHPp8TqqgLLXuM6HhLvsvURuhoJw6F/nOX6qOQWEtjcMyYp
> omShkDSLnPjs/0Iwf9vNocT7K7Sn3Gk0hOj6+ICW7wchyug8JYtuiHunP8pYrpzW
> G6l2qHMRqRs5mSENY/uWwH9qh6Z6jcfDoDDDKRTNBe0z67FzwMnX1IYCUA6XOBsZ
> iRdXe8E0CIgio+ek8HVzRm5sUlkRyfJpTXJj+pemVJhTrNCCbMGTHxzADU4Ngc8S
> +JQ+G6bsHz9R4pffsuzYFbL0avK0mm5SrjCIatE7MX171dQJ1cKpju+fAmnwuNg=
> =EAzG
> -----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/