Re: SMP interrupt patches..

Linus Torvalds (
Wed, 21 Jan 1998 15:02:41 -0800 (PST)

On Wed, 21 Jan 1998, MOLNAR Ingo wrote:
> [ this figure looks just fine on
> a 80x25 console, not sure about
> xterm ]

Looks fine.. Except I do believe that the IO-APIC is between the PCI slots
and the 8259. I don't have any schematics here, so you may well be right.

> __[ HW NMI line ] | real ISA SLOT 0-15 |
> [ SMI line ]_ | --------------------
> | | | | | | | | | | |
> /---------\ PCI _|_|_|_|_|_|_|_|_|_
> | |------[ SLOT 0 ]------PIRQ1-| |
> | |------[ SLOT 1 ]------PIRQ2-| ISA-PCI |
> | |------[ SLOT 2 ]------PIRQ3-| bridge |
> | |------[ SLOT 3 ]------PIRQ4-| |
> | IO-APIC | | |
> | |------- ISA 0 --------------| |
> | |------- ISA 1 --------------| |
> | | .... | |
> | |------- ISA 15 -------------|___________________|
> \_________/ | | | | ||||
> | | | | | | ||||____
> | | _|_|_|_|_ | |
> | | | | | slave |
> | | | 8259A |-| 8259A |
> | | | PIC | |________|
> | | |_________|
> | | |
> APIC bus INTR line INTR | ___________
> | |_______________________________________|___| |
> | |_____ CPU |
> | |local| |
> |_________________________________________________|APIC | |
> |_____|_____|

The problem is really that there is not a 1:1 mapping of PCI interrupts to
the PIRQ lines. In particular, the "IntA" interrupt on one PCI slot is not
at all necessarily mapped to the same PIRQ as "IntA" on another PCI slot.

And the problem here is that while we can query the PCI cards about the
interrupt they use, we have no way of mapping that card interrupt into the
PIRQ line it is connected to.

Usually the mapping is something fairly simple like

PIRQ = (<card interrupt> + <slot number>) % 4

but there can be an offset there, and if there are multiple PCI buses with
PCI-PCI bridges in between then that also tends to change the PIRQ.

> ISA irqs
> ========
> the ISA bus is only connected to the ISA-PCI bridge chipset.. The chipset
> 'maps' ISA interrupts to it's outgoing pins, and presents them to both the
> 8259A PIC, and to the IO-APIC.

We hope. That is probably true of all true ISA card interrupts, but it
might not be true of all motherboard interrupts on all machines. I would
not be in the least surprised if motherboard interrupts sometimes are
connected to only the 8259A PIC. This is especially true of SuperIO chips
that have both the ISA PIC _and_ the controller on-chip, so that the thing
is purely local to the chip (this tends to be floppy, printer, serial
lines, timers etc etc).

In short, we'd better default it to use the 8259A until we _know_ that the
thing is routed externally to the IO-apic too..

> PCI irqs
> ========
> PCI slots are both connected to the IO-APIC, and to the ISA-PCI bridge. The
> 4 incoming lines from PCI slots 0-3 to the ISA-PCI bridge are traditionally
> called PIRQ1-4. These 4 lines are then redirected by the bridge, and are
> presented to the 8259A PIC as ordinary 'ISA' interrupts. How to access this
> mapping info is chipset-specific.

No, each PCI slot has 4 different IRQ lines (IntA - IntD), although
usually only one is actually used (and almost always it is IntA, although
I think anything that claims to be PCI-compliant has to be able to change
it with the PCI config space).

They ALL map to PIRQ1-4, but they are pertubed depending on the PCI slot
the thing is in. So for example, in slot 0 the mapping may be the obvious
one (IntA -> PIRQ0, IntB -> PIRQ1, IntC -> PIRQ2, IntD -> PIRQ3), but in
slot 1 the mapping may be off-by-one (IntA -> PIRQ1 etc, modulo 4).

The above is done so that when you have multiple PCI cards that all
default to IntA, they don't actually use the same physical interrupt line.

And while there seems to be some common ways of mapping the card interrupt
(Int A-D) to the interrupt that the controller sees (PIRQ 0-3), it really
isn't standardized as far as I can tell: it depends on who did the board

And that's really the problem here: we can ask the BIOS how the mapping
goes, but whether we get the right answer is not necessarily dependable.
The MP table is definitely not dependable.

> which pin is which PCI IRQ, it's hard. I've seen mptables from boxes with
> two PCI buses, almost all 'high 8' IO-APIC pins were used.

Right. You get 4 physical lines per PCI slot, so in theory you only need 2
slots to be able to fill up 8 IO-APIC pins. In real life, the 4 physical
lines are shared across the whole PCI bus, and only if you have two
physically separate PCI buses do you get 8 interrupt lines.

> i've seen boxes which have the aic7xxx card on IO-APIC pin 16 (first 'high'
> pin), but the aic7xxx.c driver detects IRQ11, look at "aic7xxx.c" line 5861:
> irq = inb(INTDEF + base) & 0x0F;
> the _card firmware_ says it's IRQ 11 ... no idea how they do this, is this
> some PCI magic?

No. The card firmware actually only really says that it is "Pin 1", by
which it means that it is physically using the "IntA" line. This is really
the only thing the card itself knows.

The thing that says "11" is really only a read-write configuration space
location filled in by the BIOS setup (or not filled in by the BIOS - there
have been BIOSes where that particular piece of data wasn't very reliable
at all).

So the BIOS setup essentially knows the (hardcoded by the motherboard)
mapping of slots and card interrupt lines, and thus can tell which PIRQ
the thing really maps to physically on the IO-APIC. The BIOS has further
set up the IO-APIC to map this to the traditional IRQ 11.

> it's reasonably safe to use the 'PIRQs are connected from pin 16, consecutive'
> approach, _but_, i've seen boxes where the physical layout of PCI cards mixes
> up this order.

Right. See above on why this happens.

> conclusion: i need more info to get autodetect working, we have all the code
> in there, _some_ combination has to work.

The problem really is that it is extremely hard to autodetect how the
physical wires are set up. Anything else we can get from the PCI stuff
itself (including how a PCI-PCI bridge will change the mappings: I think
it is well-defined).

> we could get things very safe if we simply enable all IO-APIC 'high' pins
> and vector them to IRQ16-23. This means we have to autodetect IRQs in
> PCI drivers to be absolutely sure. This approach was a 100% success for
> all beta-testers who didnt mind hacking aic7xxx.c (and tulip.c, and...)
> to use the correct IRQ lines (eg use irq=17 in the above line).

Yes. I've done some of this in my current pre-81, which essentially
changes the "io_apic_irqs" initialization to include only the high
interrupts (ie 0xff0000). But I haven't changed the real IO-APIC stuff.

I'll make a pre-2.1.81 shortly that boots on everything, but only because
it defaults to the traditional APIC. It still has the code to handle the
IO-APIC, but it is essentially disabled (although it could be easily
re-enabled by (a) enabling the 1:1 mapping of the lines 16-24 and (b)
making autodetect use the PCI thing if possible..