[RFC] x86: Moving edge interrupt affinity in process context

[Thomas Gleixner]

Hi!

I'm looking into the constraint of moving interrupt affinities in interrupt
context only which we have for non remapped interrupts.

After staring into manuals and application notes for quite a while I came
to the conclusion that this is only required for level type interrupts. The
reason for this is the Remote IRR bit in the IOAPIC. From the SDM:

 "This bit is used for level triggered interrupts. Its meaning is undefined
  for edge triggered interrupts. For level triggered interrupts, this bit
  is set to 1 when local APIC(s) accept the level interrupt sent by the
  IOAPIC. The Remote IRR bit is set to 0 when an EOI message with a
  matching interrupt vector is received from a local APIC."

Moving the interrupt affinity in non interrupt context might cause a stale
remote IRR in the following case:

  IOAPIC				APIC

   pin A triggers
   sends message vector X
					accepts message, sets APIC IRR bit for
					vector X
   sets IOAPIC remote IRR bit

--> change vector for pin A to
    vector Y

					interrupt handler runs

					EOI() sends EOI message for vector X

===> Does not clear the pending remote IRR bit in the IOAPIC because vector
     X != vector Y, which is now associated to pin A, which makes that
     thing stale.

That's at least how I understand the issue from the not really elaborative
documentation.

Though for edge interrupts I'm not really understanding why we would have
to move them in interrupt context. EOI is not sent out to the originating
device for edge type interrupts, as there is no related state in the
device. Of course we'd need to keep the previous vector around and active
up to the point where the next interrupt is delivered or any other
synchronization point makes sure that it cannot be on flight to the
previous destination. But that's something we need to do anyway.

Lifting the 'only move in interrupt context' restriction would remove quite
some work from interrupt context in case of changing the interrupt affinity
of an edge interrupt, which is the vast majority of interrupts today
(MSI/MSI-X).

I started to look into that initialy to check the feasability of multi-MSI
support w/o interrupt remapping, but supporting that has other implications
as we'd need to support moving affinity of all MSI vectors in one go, which
requires rather intrusive changes to both x86 and the core interrupt
code. Not sure, that this is really worth the trouble.

It still needs to be analyzed how much gain this really brings, but I won't
start with looking into that without confirmation from the HW folks that
this is actually a safe thing to do. Even if it turns out that it's not
worth it, it would at least be good to have it documented how all of this
works and why we are using the 'move in interrupt context' mode
unconditionally.

Thanks,

	tglx