Re: a joint letter on low latency and Linux

From: Richard Gooch (rgooch@ras.ucalgary.ca)
Date: Mon Jul 03 2000 - 12:55:22 EST


Paul Barton-Davis writes:
> >> If this turns out to be true, I see it as pretty sad commentary on
> >> the state of the kernel. If its sprinkled all over with random hacks
> >> that cause it to avoid a return to user space for inordinate amounts
> >> of time, that doesn't seem to be a much better situation than the
> >> one in which there are a bunch of random hacks that seek to reverse
> >> it.
> >
> >Argghhh! It's *NOT* sprinkled all over with random hacks which prevent
> >it from scheduling early. It's a clean design with the absence of
> >random hacks.
> >
> >What you're asking for is the *introduction* of random hacks. Not the
> >reversal of random hacks. Hasn't this been made clear by now?
> >Is this an attempt at Newspeak or trying to snow people over with FUD?
>
> no, its a (failed) attempt at greco-roman rhetoric. i was just trying
> to make an allusion to the fact that nobody can point to any
> particular place in the kernel that prevents a SCHED_FIFO runnable
> thread from making back onto the processor quickly. in fact, most
> people who know about such things believe that fixing it given the
> current kernel would require patching all kinds of "random" locations
> where something is done to prevent this (presumably desirable and
> correct) behaviour from happening.
>
> so somehow, despite the fact that it would seem extremely desirable
> that if current->need_resched is set, the processor should be context
> switching ASAP, we've ended up with a system where that does not
> happen. no, i know its not because of a bunch of random hacks. i'm
> just not good enough with rhetoric to have gotten the feel for this
> across.

Let me point out a couple of examples which show just how difficult
this "fast scheduling" business is, and why we should leave these
things to RTLinux instead:

- some (broken) motherboard/IDE chipsets will lock up hard if you take
  another interrupt while servicing the IDE interrupt. Hence, the
  default IDE driver does/did disable IRQs in its interrupt
  handler. This doesn't just foul up process latencies, it can kill
  fast drivers as well

- it takes ~1 ms to read a full-sized packet from the Rx buffer of an
  NE2000 card (even on PCI!). At the very least, this will hurt
  process latencies, and will even hurt other IRQ (fast driver)
  latencies when the network driver disables IRQs

- apparently we have an O(n^3) file locking algorithm (or is it now
  O(n^2) now?). This will hurt process latencies, since we're not
  going to insert reschedule calls inside this algorithm (we're
  holding other locks at the time!)

and there will be many more examples like this. Even if you went
through and audited the kernel for sub-optimal algorithms and fixed
them (a good idea anyway), you still haven't solved the problem. And
it would be an awful lot of work. It would take man-years just for the
core kernel code (ignoring device drivers).

There would remain the problem of naughty device drivers. In some
cases, it's a matter of bad (read: selfish) programming, and of course
we'd like that fixed. But there are a lot of device drivers in the
kernel. It would take even longer to audit all these.

Even after fixing selfish drivers, there's still the problem of
devices where you have to disable IRQs, or IRQ processing takes a long
time. We can't help this, since Linux is a general-purpose OS that has
to work for all hardware.

The way RTLinux gets around these kinds of problems is that it steals
control of interrupt management from Linux. That means it won't work
with some (broken) hardware. And since it doesn't ship with a large
number of device drivers, it's pretty easy to get an idea what
latencies can be expected. Linux, with its thousands of device
drivers, makes this a pretty daunting task. How do you distinguish
between a lazy driver which disables IRQs and one which is dealing
with broken hardware (say for the 1% of cards which have a particular
lockup problem)?

And you can't just say "we just need a few tweaks here and there".
Without auditing, we just don't know what kind of latencies to
expect. Maybe you can hack up your kernel then run tests on your
hardware with your typical load and decide if you're happy, but that's
not a general solution. It's not even sufficient for people running
the same class of soft-RT applications as you, because their hardware
and typical load will be different.

                                Regards,

                                        Richard....
Permanent: rgooch@atnf.csiro.au
Current: rgooch@ras.ucalgary.ca

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Fri Jul 07 2000 - 21:00:13 EST