Re: [PATCH RFC 2/2] irqchip/gic: Allow the use of SGI interrupts

[Florian Fainelli]

Hello marc,

On 10/23/19 6:22 AM, Marc Zyngier wrote:
> Hi Florian,
> 
> Needless to say, I mostly have questions...
> 
> On 2019-10-23 01:05, Florian Fainelli wrote:
>> SGI interrupts are a convenient way for trusted firmware to target a
>> specific set of CPUs. Update the ARM GIC code to allow the translation
>> and mapping of SGI interrupts.
>>
>> Since the kernel already uses SGIs for various inter-processor interrupt
>> activities, we specifically make sure that we do not let users of the
>> IRQ API to even try to map those.
>>
>> Internal IPIs remain dispatched through handle_IPI() while public SGIs
>> get promoted to a normal interrupt flow management.
>>
>> Signed-off-by: Florian Fainelli <f.fainelli@xxxxxxxxx>
>> ---
>> Âdrivers/irqchip/irq-gic.c | 41 +++++++++++++++++++++++++++------------
>> Â1 file changed, 29 insertions(+), 12 deletions(-)
>>
>> diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
>> index 30ab623343d3..dcfdbaacdd64 100644
>> --- a/drivers/irqchip/irq-gic.c
>> +++ b/drivers/irqchip/irq-gic.c
>> @@ -385,7 +385,10 @@ static void __exception_irq_entry
>> gic_handle_irq(struct pt_regs *regs)
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ * Pairs with the write barrier in gic_raise_softirq
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ */
>> ÂÂÂÂÂÂÂÂÂÂÂÂ smp_rmb();
>> -ÂÂÂÂÂÂÂÂÂÂÂ handle_IPI(irqnr, regs);
>> +ÂÂÂÂÂÂÂÂÂÂÂ if (irqnr < NR_IPI)
>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ handle_IPI(irqnr, regs);
>> +ÂÂÂÂÂÂÂÂÂÂÂ else
>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ handle_domain_irq(gic->domain, irqnr, regs);
> 
> Double EOI, UNPREDICTABLE territory, your state machine is now dead.

Oh yes, the interrupt flow now also goes through ->irq_eoi (that's the
whole point), meh.

> 
>> Â#endif
>> ÂÂÂÂÂÂÂÂÂÂÂÂ continue;
>> ÂÂÂÂÂÂÂÂ }
>> @@ -1005,20 +1008,34 @@ static int gic_irq_domain_translate(struct
>> irq_domain *d,
>> ÂÂÂÂÂÂÂÂ if (fwspec->param_count < 3)
>> ÂÂÂÂÂÂÂÂÂÂÂÂ return -EINVAL;
>>
>> -ÂÂÂÂÂÂÂ /* Get the interrupt number and add 16 to skip over SGIs */
>> -ÂÂÂÂÂÂÂ *hwirq = fwspec->param[1] + 16;
>> -
>> -ÂÂÂÂÂÂÂ /*
>> -ÂÂÂÂÂÂÂÂ * For SPIs, we need to add 16 more to get the GIC irq
>> -ÂÂÂÂÂÂÂÂ * ID number
>> -ÂÂÂÂÂÂÂÂ */
>> -ÂÂÂÂÂÂÂ if (!fwspec->param[0])
>> +ÂÂÂÂÂÂÂ *hwirq = fwspec->param[1];
>> +ÂÂÂÂÂÂÂ switch (fwspec->param[0]) {
>> +ÂÂÂÂÂÂÂ case 0:
>> +ÂÂÂÂÂÂÂÂÂÂÂ /*
>> +ÂÂÂÂÂÂÂÂÂÂÂÂ * For SPIs, we need to add 16 more to get the GIC irq
>> +ÂÂÂÂÂÂÂÂÂÂÂÂ * ID number
>> +ÂÂÂÂÂÂÂÂÂÂÂÂ */
>> +ÂÂÂÂÂÂÂÂÂÂÂ *hwirq += 16;
>> +ÂÂÂÂÂÂÂÂÂÂÂ /* fall through */
>> +ÂÂÂÂÂÂÂ case 1:
>> +ÂÂÂÂÂÂÂÂÂÂÂ /* Add 16 to skip over SGIs */
>> ÂÂÂÂÂÂÂÂÂÂÂÂ *hwirq += 16;
>> +ÂÂÂÂÂÂÂÂÂÂÂ *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
>>
>> -ÂÂÂÂÂÂÂ *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
>> +ÂÂÂÂÂÂÂÂÂÂÂ /* Make it clear that broken DTs are... broken */
>> +ÂÂÂÂÂÂÂÂÂÂÂ WARN_ON(*type == IRQ_TYPE_NONE);
>> +ÂÂÂÂÂÂÂÂÂÂÂ break;
>> +ÂÂÂÂÂÂÂ case 2:
>> +ÂÂÂÂÂÂÂÂÂÂÂ /* Refuse to map internal IPIs */
>> +ÂÂÂÂÂÂÂÂÂÂÂ if (*hwirq < NR_IPI)
> 
> So depending on how the kernel uses SGIs, you can or cannot use these SGIs.
> That looks like a good way to corner ourselves into not being to change
> much.

arch/arm/kernel/smp.c has a forward looking statement about SGI numbering:

        /*
         * SGI8-15 can be reserved by secure firmware, and thus may
         * not be usable by the kernel. Please keep the above limited
         * to at most 8 entries.
         */

is this something that can be used as an universal and unbreakable rule
for the ARM64 kernel as well in order to ensure SGIs 8-15 can be usable
through the IRQ API or is this simply not a guarantee at all?

> 
> Also, do you expect this to work for both Group-0 and Group-1 interrupts
> (since you imply that this works as a communication medium with the secure
> side)? Given that the kernel running in NS has no way to enable/disable
> Group-0 interrupts, this looks terminally flawed. Or is that Group-1 only?

That would be Group-1 interrupts only, are you suggesting there is an
additional check being done that such SGIs are actually part of Group-1?

> 
> How do we describe which SGIs are guaranteed to be available to Linux?

In our case, the Device Tree mailbox node gets populated its interrupts
property with the SGI number(s), and that same number is also passed as
a configuration parameter to the trusted firmware. Or are you echoing
back to your earlier comment about the fact that if the kernel changes
its own definition of NR_IPI then we suddenly start breaking IRQ API
uses of SGIs in a certain range?

> 
>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ return -EPERM;
>> +
>> +ÂÂÂÂÂÂÂÂÂÂÂ *type = IRQ_TYPE_NONE;
> 
> Or not. SGI are edge triggered, by definition.
> 
>> +ÂÂÂÂÂÂÂÂÂÂÂ break;
>> +ÂÂÂÂÂÂÂ default:
>> +ÂÂÂÂÂÂÂÂÂÂÂ break;
>> +ÂÂÂÂÂÂÂ }
>>
>> -ÂÂÂÂÂÂÂ /* Make it clear that broken DTs are... broken */
>> -ÂÂÂÂÂÂÂ WARN_ON(*type == IRQ_TYPE_NONE);
> 
> Really?

Given the comment in gic_set_type() about SGIs, the WARN_ON() was moved
above to continue checking for GIC_SPI and GIC_PPI, but we should
extract the type from the Devic eTree and only permit an edge mask.
-- 
Florian