Re: [PATCH] x86, Fix do_IRQ interrupt warning for cpu hotplug retriggeredirqs
From: Prarit Bhargava
Date: Mon Dec 23 2013 - 10:29:50 EST
On 12/23/2013 04:41 AM, rui wang wrote:
> On 12/2/13, Prarit Bhargava <prarit@xxxxxxxxxx> wrote:
>> Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=64831
>>
>> When downing a cpu it is possible that there are unhandled irqs left in
>> the APIC IRR register. fixup_irqs() goes through the IRR and retriggers
>> the IRQs left in the APIC IRR. After this, the vector for the irq is set
>> to -1. There is a possibility here, however, that the CPU does handle an
>> irq in the IRR and then calls the vector.
>>
>
> The patch does not seem to root-cause the problem. It seems to hide
> the real problem.
>
> It is not possible that a device-triggered irq can arrive to this cpu
> again after fixup_irqs() fills its vector_irq[vector] to -1, because
> we've done the following:
>
> 1. We disabled interrupt on this cpu in stop_machine().
> 2. We called irq_set_affinity() to exclude this cpu as a target for the irq.
> 3. We checked APIC_IRR and re-triggered any pending irqs to other cpus.
... and we set the IRQ handler to -1 for the down'd cpu.
Rui, I think you're right up to here but I think this has nothing to do with IPI
or locking.
I assumed that the issue I was trying to fix was long standing and well-known
within the kernel given some of the comments I had read here-and-there about
people seeing the do_IRQ errors on LKML. There have long been reports of the
do_IRQ warning output during cpu down.
Here's what the issue is after step 3 above...
4. The APIC_IRR is still *set* in the down'd cpu with IRQs disabled.
5. We continue executing the stop_machine "down" portion of the code, then
continue executing in normal context the "die" code (ie, __cpu_die()).
IRQ disable only pertains stop_machine down. So after we leave that context,
IRR will still execute. While the kernel is spinning in cpu_die(), the down'd
cpu attempts to execute handler for IRQ in IRR ... and can't find one because
we've set it to -1. So we see the warning.
A few additional debug points:
1. I put a printk in fixup_irq when we call the irq_retrigger on another cpu
that dumps the the down'd CPU and IRQ # in fixup_irqs(). I see that printk
*EVERYTIME* I see the do_IRQ warning.
2. The do_IRQ warning *always* appears before I see the offline message ...
[ 148.656016] Broke affinity for irq 634
[ 148.660493] Broke affinity for irq 698
[ 148.665739] kvm: disabling virtualization on CPU58
[ 148.666732] PRARIT: 58.208 IRR entry ... irq_retrigger call.
at this point we've left the stop_machine() code and we're now continuing to
execute ... then we hit the cpu_die() ... which spins.
[ 148.671106] do_IRQ: 58.208 No irq handler for vector (irq -1)
[ 148.677544] smpboot: CPU 58 is now offline
I think I have root caused this to the IRR being set in the down'd cpu. It is
admittedly a rare occurrence in the kernel. I usually have to run about 1000 up
and down's before hitting it, however, on my current test system it seems to hit
much more frequently, almost 1 in 64 times.
P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/