Re: x86: interrupt routing question

From: Mike Galbraith
Date: Wed Sep 28 2011 - 03:28:58 EST


For the next sod who gets curious about irq routing behavior difference,
and tries to ask google.

On Tue, 2011-07-12 at 09:27 +0200, Mike Galbraith wrote:
> Greetings,
>
> I have an x3550 M3 box which shows different behavior than my Q6600
> desktop box. The below is the rt kernel, but it doesn't matter which
> kernel I boot, all interrupts are on CPU0 unless I move them. On Q6600
> box, IRQs magically appear on every cpu in the affinity mask. Change
> the eth0 mask while flood pinging, numbers start/stop changing on the
> fly.
>
> On x3550 M3, setting eg IRQ 63's mask to '0xe' will move irq to CPU2,
> but nothing that I have found will make the thing behave the same as
> trusty old Q6600 box, which seems strange. Is this some kind of BIOS
> thingie, or is every kernel busted on this hardware? Busted kernel
> seems highly doubtful.

The difference is that Q6600 desktop box uses flat routing, and x3550 M3
uses physical flat. What the heck does that mean you (didn't) ask?
Dunno, that's a "RTFM for all the gory technical details" thing, but if
you rummage around in x86 source, you'll end up here..

arch/kernel/x86/apic/io_apic.c::setup_ioapic_irq()

if (assign_irq_vector(irq, cfg, apic->target_cpus()))
return;

dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());

apic_printk(APIC_VERBOSE,KERN_DEBUG
"IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> "
"IRQ %d Mode:%i Active:%i Dest:%d)\n",
apic_id, mpc_ioapic_id(apic_id), pin, cfg->vector,
irq, trigger, polarity, dest);

..and see that in physical flat mode, assign_irq_vector() scribbles only
one bit to cfg->domain.

arch/kernel/x86/apic/io_apic.c:__assign_irq_vector()

for_each_cpu_and(cpu, mask, cpu_online_mask) {
int new_cpu;
int vector, offset;

apic->vector_allocation_domain(cpu, tmp_mask);
...
cpumask_copy(cfg->domain, tmp_mask);
err = 0;
break;
}

apic.vector_allocation_domain for physical flat is..

arch/x86/kernel/apic/apic_flat_64.c::

static void physflat_vector_allocation_domain(int cpu, struct cpumask *retmask)
{
cpumask_clear(retmask);
cpumask_set_cpu(cpu, retmask);
}

..whereas for flat it's..

static void flat_vector_allocation_domain(int cpu, struct cpumask *retmask)
{
...
cpumask_clear(retmask);
cpumask_bits(retmask)[0] = APIC_ALL_CPUS;
}

So, while __assign_irq_vector() is called with a mask of all CPUs for
both boxen, apic->vector_allocation_domain() ensures the mask finally
written to cfg->domain has only one CPU bit set in physical flat mode,
and all CPU bits set in flat mode.

-Mike

P.S. if you try that RTFM thing, strap a pillow to your forehead first

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/