Re: Interrupt Affinity in SMP

From: Bryan Hundven
Date: Sun Jul 18 2010 - 14:52:58 EST


On Sun, Jul 18, 2010 at 11:38 AM, Ciju Rajan K <ciju@xxxxxxxxxxxxxxxxxx> wrote:
> Bryan Hundven wrote:
>>
>> On Sat, Jul 10, 2010 at 6:20 PM, Robert Hancock <hancockrwd@xxxxxxxxx>
>> wrote:
>>
>>>
>>> On Sat, Jul 10, 2010 at 1:46 PM, Bryan Hundven <bryanhundven@xxxxxxxxx>
>>> wrote:
>>>
>>>>
>>>> I was able to set eth0 and it's TxRx queues to cpu1, but it is my
>>>> understanding that 0xFFFFFFFF should distribute the interrupts across
>>>> all
>>>> cpus, much like LOC in my output of /proc/interrupts.
>>>>
>>>> I don't have access to the computer this weekend, but I will provide
>>>> more
>>>> info on Monday.
>>>>
>>>
>>> That may be chipset dependent, I don't think all chipsets have the
>>> ability to distribute the interrupts like that. Round-robin interrupt
>>> distribution for a given handler isn't optimal for performance anyway
>>> since it causes the relevant cache lines for the interrupt handler to
>>> be ping-ponged between the different CPUs.
>>>
>>>
>>>>
>>>> -bryan
>>>>
>>>> On Jul 9, 2010 5:48 PM, "Robert Hancock" <hancockrwd@xxxxxxxxx> wrote:
>>>>
>>>> On 07/09/2010 04:59 PM, Bryan Hundven wrote:
>>>>
>>>>>
>>>>> Mauro, list,
>>>>>
>>>>> (please CC me in replies, I am not...
>>>>>
>>>>
>>>> Tried changing these files to exclude CPU0?
>>>>
>>>> Have you tried running the irqbalance daemon? That's what you likely
>>>> want to
>>>> be doing anyway..
>>>>
>>>>
>>>>>
>>>>> =====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<=====
>>>>>
>>>>> =====8<=====8<=====8<==...
>>>>>
>>
>> Please see the two attached examples.
>>
>> Notice on the 5410 example how we start with the affinity set to 0xff,
>> and change it to 0xf0.
>> This should spread the interrupts over the last 4 cores of this quad
>> core - dual processor system.
>>
>> Also notice on the 5645 example, with the same commands we start with
>> 0xffffff and change to 0xfff000 to spread the interrupts over the last
>> 12 cores, but only the first of the last twelve cores receive
>> interrupts.
>>
>> This is the inconsistency I was trying to explain before.
>>
>
> What was the status of irqbalance daemon? Was it turned on? If it is
> running, there is a chance that the interrupt count is within the threshold
> limit and interrupts are not being routed to the other core.

irqbalance daemon was not running on either setup.

> Could you also try with increasing the interrupt load and see if the
> distribution is happening among the cores?

We use spirent testcenter l2/l3 test equipment and pushed 100%
throughput with the same distribution. Nothing changed.

This isn't affecting just ethernet drivers. I have also seen the same
issues with hardware encryption devices and other hardware that gets a
software interrupt.

--Bryan

>
> -Ciju
>>
>> --Bryan
>>
>
>



--
Bryan Hundven
bryanhundven@xxxxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/