Re: [GIT PULL] Introduce housekeeping subsystem v4

[Chris Metcalf]

On 10/20/2017 10:29 AM, Frederic Weisbecker wrote:
> 2017-10-20 10:17 UTC+02:00, Ingo Molnar <mingo@xxxxxxxxxx>:
> > I mean code like:
> >
> >   triton:~/tip> git grep on_each_cpu mm
> >   mm/page_alloc.c:         * cpu to drain that CPU pcps and on_each_cpu_mask
> >   mm/slab.c:      on_each_cpu(do_drain, cachep, 1);
> >   mm/slub.c:      on_each_cpu_cond(has_cpu_slab, flush_cpu_slab, s, 1,
> > GFP_ATOMIC);
> >   mm/vmstat.c:    err = schedule_on_each_cpu(refresh_vm_stats);
> >
> > is something we want to execute on 'housekeeping CPUs' as well, to not
> > disturb the
> > isolated CPUs, right?
> I see, so indeed that's the kind of thing we want to also confine to
> housekeeping as
> well whenever possible but these cases require special treatment that need to be
> handled by the subsystem in charge. For example vmstat has the vmstat_sheperd
> thing which allows to drive those timers adaptively on demand to make sure that
> userspace isn't interrupted. The others will likely need some similar treatment.
>
> For now I only see vmstat having such a feature and it acts
> transparently. There is
> also the LRU flush (IIRC) which needs to be called for example before
> returning to
> userspace to avoid IPIs. Such things may indeed need special treatment. With the
> current patchset it could be a housekeeping flag.

I have been working to update the task isolation support the last few
days and though it's not quite ready to post (probably will be Monday
or Tuesday), I have sorted out those issues from task isolation's
perspective. It turns out that you can both quiesce the
vmstat_shepherd, as well as drain the LRU per-cpu pages, while
interrupts are disabled on the way back to userspace.

Whether shifting this work to housekeeping cores at all times makes
sense seems like a much more open question. The idea of task
isolation is to provide a harder guarantee of isolation, and in
particular to shift work to the moment that you return to userspace,
rather than allowing it to happen later. It does seem likely that
there are some things you'd want to do on the core itself most of the
time, and just suppress for true task isolation if requested, rather
than trying to move them to the housekeeping cores.

But, it's certainly worth looking at both options and seeing how it
plays out. The less complicated the task isolation return-to-user
path is, the better. (The idea of task isolation seems like a win no
matter what, to allow ensuring kernel isolation when you absolutely
require it.)

The current task isolation tree is in the "dataplane" branch at
git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile.git

--
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com