Re: [EXT] Re: [PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks

[Alex Belits]

On Thu, 2020-10-01 at 16:47 +0200, Frederic Weisbecker wrote:
> External Email
> 
> -------------------------------------------------------------------
> ---
> On Wed, Jul 22, 2020 at 02:58:24PM +0000, Alex Belits wrote:
> > From: Yuri Norov <ynorov@xxxxxxxxxxx>
> > 
> > If CPU runs isolated task, there's no any backlog on it, and
> > so we don't need to flush it.
> 
> What guarantees that we have no backlog on it?

I believe, the logic was that it is not supposed to have backlog
because it could not be produced while the CPU was in userspace,
because one has to enter kernel to receive (by interrupt) or send (by
syscall) anything.

Now, looking at this patch. I don't think, it can be guaranteed that
there was no backlog before it entered userspace. Then backlog
processing will be delayed until exit from isolation. It won't be
queued, and flush_work() will not wait when no worker is assigned, so
there won't be a deadlock, however this delay may not be such a great
idea.

So it may be better to flush backlog before entering isolation, and in
flush_all_backlogs() instead of skipping all CPUs in isolated mode,
check if their per-CPU softnet_data->input_pkt_queue and softnet_data-
>process_queue are empty, and if they are not, call backlog anyway.
Then, if for whatever reason backlog will appear after flushing (we
can't guarantee that nothing preempted us then), it will cause one
isolation breaking event, and if nothing will be queued before re-
entering isolation, there will be no backlog until exiting isolation.

> 
> > Currently flush_all_backlogs()
> > enqueues corresponding work on all CPUs including ones that run
> > isolated tasks. It leads to breaking task isolation for nothing.
> > 
> > In this patch, backlog flushing is enqueued only on non-isolated
> > CPUs.
> > 
> > Signed-off-by: Yuri Norov <ynorov@xxxxxxxxxxx>
> > [abelits@xxxxxxxxxxx: use safe task_isolation_on_cpu()
> > implementation]
> > Signed-off-by: Alex Belits <abelits@xxxxxxxxxxx>
> > ---
> >  net/core/dev.c | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 90b59fc50dc9..83a282f7453d 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -74,6 +74,7 @@
> >  #include <linux/cpu.h>
> >  #include <linux/types.h>
> >  #include <linux/kernel.h>
> > +#include <linux/isolation.h>
> >  #include <linux/hash.h>
> >  #include <linux/slab.h>
> >  #include <linux/sched.h>
> > @@ -5624,9 +5625,13 @@ static void flush_all_backlogs(void)
> >  
> >  	get_online_cpus();
> >  
> > -	for_each_online_cpu(cpu)
> > +	smp_rmb();
> 
> What is it ordering?

Same as with other calls to task_isolation_on_cpu(cpu), it orders
access to ll_isol_flags.

> > +	for_each_online_cpu(cpu) {
> > +		if (task_isolation_on_cpu(cpu))
> > +			continue;
> >  		queue_work_on(cpu, system_highpri_wq,
> >  			      per_cpu_ptr(&flush_works, cpu));
> > +	}
> >  
> >  	for_each_online_cpu(cpu)
> >  		flush_work(per_cpu_ptr(&flush_works, cpu));
> 
> Thanks.