Re: [PATCH 3/6] dataplane nohz: run softirqs synchronously on user entry

[Mike Galbraith]

On Mon, 2015-05-11 at 16:13 -0400, Chris Metcalf wrote:
> On 05/09/2015 03:04 AM, Mike Galbraith wrote:
> > On Fri, 2015-05-08 at 13:58 -0400, Chris Metcalf wrote:
> >> For tasks which have elected dataplane functionality, we run
> >> any pending softirqs for the core before returning to userspace,
> >> rather than ever scheduling ksoftirqd to run.  The problem we
> >> fix is that by allowing another task to run on the core, we
> >> guarantee more interrupts in the future to the dataplane task,
> >> which is exactly what dataplane mode is required to prevent.
> > If ksoftirqd were rt class
> 
> I realize I actually don't know if this is true or not.  Is
> ksoftirqd rt class?  If not, it does seem pretty plausible that
> it should be...

It is in an rt kernel, not in a stock kernel, it's malleable in both ;-)

> > softirqs would be gone when the soloist gets
> > the CPU back and heads to userspace.  Being a soloist, it has no use for
> > a priority, so why can't it just let ksoftirqd run if it raises the
> > occasional softirq?  Meeting a contended lock while processing it will
> > wreck the soloist regardless of who does that processing.
> 
> The thing you want to avoid is having two processes both
> runnable at once, since then the "quiesce" mode can't make
> forward progress and basically spins in cpu_idle() until ksoftirqd
> can come in.

The only way ksoftirqd can appear is the soloist woke it.  If alleged
soloist is raising enough softirqs to matter, it ain't really an ultra
sensitive solo artist, it's part of a noise inducing (locks) chorus.

>   Alas, my recollection of the precise failure mode
> is somewhat dimmed; my commit notes from a year ago (for
> a variant of the patch I'm upstreaming now):
> 
>          - Trying to return to userspace with pending softirqs is not
>            currently allowed.  Prior to this patch, when this happened
>            we would just wait in cpu_idle.  Instead, what we now do is
>            directly run any pending softirqs, then go back and retry the
>            path where we return to userspace.
>          
>          - Raising softirqs (in this case for hrtimer support) could
>            cause the ksoftirqd daemon to be woken on a core.  This is
>            bad because on a dataplane core, a QUIESCE process will
>            then block until the ksoftirqd runs, and the system sometimes
>            seems to flag that soft irqs are available but not schedule
>            the timer to arrange for a context switch to ksoftirqd.
>            To handle this, we avoid bailing out in __do_softirq() when
>            we've been working for a while, if we're on a dataplane core,
>            and just keep working until done.  Similarly, on a dataplane
>            core running a userspace task, we don't wake ksoftirqd when
>            we are raising a softirq, even if we're not in an interrupt
>            context where it will run promptly, since a non-interrupt
>            context will also run promptly.

Thomas has nuked the hrtimer softirq.

> I'm happy to drop this patch entirely from the series for now, and
> if ksoftirqd shows up as a problem going forward, we can address it
> as necessary at that time.   What do you think?

Inlining softirqs may save a context switch, but adds cycles that we may
consume at higher frequency than the thing we're avoiding.

	-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/