Re: Using irq-crossbar.c

From: Sebastian Frias
Date: Mon Jun 13 2016 - 11:46:35 EST

Hi Marc,

Thanks for your reply, please find my comments below.

On 06/10/2016 06:05 PM, Marc Zyngier wrote:
> On 10/06/16 16:37, Sebastian Frias wrote:
>> Hi,
>>
>> We are trying to write a driver for an interrupt controller (actually
>> more of a crossbar) for an ARM-based SoC. This IRQ crossbar has 128
>> inputs and 24 outputs, the outputs are connected directly to the
>> GIC. The idea is that the GIC handles everything, and just request a
>> mapping from an IRQ number (0...127, from a device's DT entry) into
>> one of its 24 input lines.
>
> "Just request a mapping...".
>
>> By looking at current code (4.7-rc1) there seems to be a driver
>> (drivers/irqchip/irq-crossbar.c) that provides similar
>> functionality. The driver uses hierarchical irq domains (since commit
>> 783d31863fb8 "irqchip: crossbar: Convert dra7 crossbar to stacked
>> domains") which we believe we don't need because the only controller
>> is the GIC.
>
> So you need it, but you don't need it? The GIC may be the only interrupt
> controller with which software interacts when the interrupt occurs, but
> the crossbar does play a major role in *routing* the interrupt to the
> right GIC pin.

My understanding of "hierarchical irq domains" is that it is useful when there are multiple stacked interrupt controllers.
Also, the documentation says "the majority of drivers should use the linear map" (as opposed to the hierarchical one).

Maybe the definition of "interrupt controller" could benefit from some clarification, but my understanding is that, in our case, the GIC is the only interrupt controller (that's where the interrupt type must be set active_high/active_low/edge, etc.), in front of it, it happens to be a crossbar, that happens to be programmable, and that can be used to map any 128 line into any of 24 lines of the GIC (actually it is a many-to-many router, without any latching nor edge detection functionality)

Obviously, when the DT says that ethernet driver uses IRQ=120 (for example), the crossbar must be setup to route line 120 to one of the 24 lines going to the GIC.
So a minimum of interaction between the GIC driver (and/or the Linux IRQ framework) and the driver programming the crossbar is required, and that's what we are trying to achieve.

Does that makes sense?

>
>> However the API used previously, register_routable_domain_ops(), was
>> removed with commit a5561c3e845c "irqchip: gic: Get rid of routable
>> domain".
>
> And every day, I thank $DEITY for having delivered us from this evil.
> Really. And it wasn't much of an API. It was the son of a hack, bolted
> on the side of another hack. Unmaintainable, getting in the way. I had
> much fun slaughtering it! ;-)
>
>> Trying to use the driver with hierarchical domains (after
>> modifications for our SoC), results on the kernel being blocked at
>> some point:
>>
>> [ 0.041524] ThumbEE CPU extension supported.
>> [ 0.041589] Registering SWP/SWPB emulation handler
>> [ 0.052022] Freeing unused kernel memory: 12364K (c029b000 - c0eae000)
>> [ 0.074084] random: dbus-uuidgen urandom read with 0 bits of entropy available
>
> Sorry, but that's not much of a log. Anything related to interrupts, maybe?

That's the last log (it's stuck there) and I was asking how/where to enable more logs to be able to debug this.

Or there are no standardised logs and every person has to come up with its own debug logs?

>
>> We've put logs on the different domain_ops calls (alloc, free,
>> translate) but they are not called, even if the DT is supposed to
>> tell devices to take interrupts from this controller (*).
>
> At all? Nobody is talking to the GIC?
>
>> Do you have suggestions on what APIs should be used, further
>> reading/examples and/or pointers on how debug this (logs to enable,
>> things to look for, etc.)?
>
> You could start with posting actual logs of an interrupt being
> requested, as well as perform some basic tracing of the various
> callbacks into the irqdomain and irqchip layers, all the way down to the
> interrupt controllers (note the plural).

Ok, thanks.

In the meanwhile, and in case irq-crossbar is not the good example for our case, would it be possible to get some guidance, examples, tips, on how to write a driver like the one described? ie:

[0..127] IRQ inputs => BIG_MUX => [0..23] outputs => [0..23] GIC inputs

BIG_MUX is a many-to-many router:
- 128x32bit registers to setup a route between any of the 128 input (IRQ_dev) and any of the 24 outputs (IRQ_gic)
- 4x32bit registers that read the RAW status of each of the 128 lines (no latching nor edge detection)
not sure how useful is to read such RAW status, because (naive hypothesis follows) Linux's IRQ framework could remember which IRQ_dev lines are routed to which IRQ_gic, and thus when handling IRQ_gic(x), Linux could just ask the drivers tied to it to check for interruptions.


>Also, without seeing the code,
> it is pretty difficult to make any meaningful comment...

Base code is either 4.7rc1 or 4.4.
The irq-crossbar code is not much different from TI, but you can find it attached.

>
> Where is the GIC? Where is the #interrupt-cells property? What is the
> interrupt parent for the GIC itself? (and I'm tempted to add "What is
> your name? What is you quest?", but that's because it is Friday and I
> feel like I need a beer...).

Anything missing on the diff can be found on the original file:
arch/arm/boot/dts/tango4-common.dtsi

Thanks in advance.
Best regards,

Sebastian

#include <linux/init.h>
#include <linux/irq.h>
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/ioport.h>
#include <linux/io.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/slab.h>


#define IRQ_MUX_INPUT_LINES (128)
#define IRQ_MUX_OUTPUT_LINES (24)
#define IRQ_FREE (-1)
#define IRQ_RESERVED (-2)


/**
* struct tangox_irq_mux : irq mux private driver data
* @lock: spinlock serializing access to @irq_map
* @mux_inputs: inputs (irq lines entering) the mux
* @mux_outputs: outputs (irq lines exiting) the mux (connected to the GIC)
* @irq_map: irq input->output map
* @reg_base: mux base address
*/
struct tangox_irq_mux {
raw_spinlock_t lock;
uint mux_inputs;
uint mux_outputs;
uint *irq_map;
void __iomem *reg_base;
};


#define DBGLOG(__format, ...) \
do { \
pr_info("[%s:%d] %s(): " __format, __FILE__, __LINE__, __FUNCTION__ , ##__VA_ARGS__); \
} while (0)


static inline u32 intc_readl(int address)
{
u32 value = readl_relaxed((void __iomem *)address);
//DBGLOG("read 0x%x @ 0x%x\n", value, address);
return value;
}

static inline void intc_writel(int value, int address)
{
//DBGLOG("write 0x%x @ 0x%x\n", value, address);
writel_relaxed(value, (void __iomem *)address);
}

static inline void tangox_setup_irq_route(struct tangox_irq_mux *irq_mux_context, int irq_in, int irq_out)
{
u32 value = irq_out;
u32 offset = (irq_in * 4);
u32 address = irq_mux_context->reg_base + offset;

DBGLOG("route irq %2d (@ 0x%08x) => irq %2d\n", irq_in, address, value);

if (value)
value |= 0x80000000;

intc_writel(value, address);
}


static int tangox_allocate_gic_irq(struct irq_domain *domain,
unsigned virq,
irq_hw_number_t hwirq)
{
struct tangox_irq_mux *irq_mux_context = domain->host_data;
struct irq_fwspec fwspec;
int i;
int err;

DBGLOG("domain 0x%p, virq %d (0x%x) hwirq %d (0x%x)\n", domain, virq, virq, hwirq, hwirq);

if (!irq_domain_get_of_node(domain->parent))
return -EINVAL;

raw_spin_lock(&(irq_mux_context->lock));
for (i = irq_mux_context->mux_outputs - 1; i >= 0; i--) {
if (irq_mux_context->irq_map[i] == IRQ_FREE) {
irq_mux_context->irq_map[i] = hwirq;
break;
}
}
raw_spin_unlock(&(irq_mux_context->lock));

if (i < 0)
return -ENODEV;

fwspec.fwnode = domain->parent->fwnode;
fwspec.param_count = 3;
fwspec.param[0] = 0; /* SPI */
fwspec.param[1] = i;
fwspec.param[2] = IRQ_TYPE_LEVEL_HIGH;

err = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);
if (err)
irq_mux_context->irq_map[i] = IRQ_FREE;
else
tangox_setup_irq_route(irq_mux_context, hwirq, i);

return err;
}

static struct irq_chip mux_chip = {
.name = "CBAR",
.irq_eoi = irq_chip_eoi_parent,
.irq_mask = irq_chip_mask_parent,
.irq_unmask = irq_chip_unmask_parent,
.irq_retrigger = irq_chip_retrigger_hierarchy,
.irq_set_type = irq_chip_set_type_parent,
.flags = IRQCHIP_MASK_ON_SUSPEND |
IRQCHIP_SKIP_SET_WAKE,
#ifdef CONFIG_SMP
.irq_set_affinity = irq_chip_set_affinity_parent,
#endif
};

/**
* tangox_irq_mux_domain_alloc - map/reserve a mux<->irq connection
* @domain: domain of irq to map
* @virq: virq number
* @nr_irqs: number of irqs to reserve
*
*/
static int tangox_irq_mux_domain_alloc(struct irq_domain *domain,
unsigned int virq,
unsigned int nr_irqs,
void *data)
{
struct tangox_irq_mux *irq_mux_context = domain->host_data;
struct irq_fwspec *fwspec = data;
irq_hw_number_t hwirq;
int i;

DBGLOG("domain 0x%p, virq %d (0x%x) nr_irqs %d\n", domain, virq, virq, nr_irqs);

if (fwspec->param_count != 3)
return -EINVAL; /* Not GIC compliant */
if (fwspec->param[0] != 0)
return -EINVAL; /* No PPI should point to this domain */

hwirq = fwspec->param[1];
if ((hwirq + nr_irqs) > irq_mux_context->mux_inputs)
return -EINVAL; /* Can't deal with this */

for (i = 0; i < nr_irqs; i++) {
int err = tangox_allocate_gic_irq(domain, virq + i, hwirq + i);

if (err)
return err;

irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
&mux_chip, NULL);
}

return 0;
}

/**
* tangox_irq_mux_domain_free - unmap/free a mux<->irq connection
* @domain: domain of irq to unmap
* @virq: virq number
* @nr_irqs: number of irqs to free
*
* We do not maintain a use count of total number of map/unmap
* calls for a particular irq to find out if a irq can be really
* unmapped. This is because unmap is called during irq_dispose_mapping(irq),
* after which irq is anyways unusable. So an explicit map has to be called
* after that.
*/
static void tangox_irq_mux_domain_free(struct irq_domain *domain,
unsigned int virq,
unsigned int nr_irqs)
{
int i;
struct tangox_irq_mux *irq_mux_context = domain->host_data;

DBGLOG("domain 0x%p, virq %d (0x%x) nr_irqs %d\n", domain, virq, virq, nr_irqs);

raw_spin_lock(&(irq_mux_context->lock));
for (i = 0; i < nr_irqs; i++) {
struct irq_data *irqdata = irq_domain_get_irq_data(domain, virq + i);

irq_domain_reset_irq_data(irqdata);
irq_mux_context->irq_map[irqdata->hwirq] = IRQ_FREE;
tangox_setup_irq_route(irq_mux_context, 0x0, irqdata->hwirq);
}
raw_spin_unlock(&(irq_mux_context->lock));
}

static int tangox_irq_mux_domain_translate(struct irq_domain *domain,
struct irq_fwspec *fwspec,
unsigned long *hwirq,
unsigned int *type)
{

DBGLOG("domain 0x%p\n", domain);

if (is_of_node(fwspec->fwnode)) {
if (fwspec->param_count != 3)
return -EINVAL;

/* No PPI should point to this domain */
if (fwspec->param[0] != 0)
return -EINVAL;

*hwirq = fwspec->param[1];
*type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
return 0;
}

return -EINVAL;
}

static const struct irq_domain_ops tangox_irq_mux_domain_ops = {
.alloc = tangox_irq_mux_domain_alloc,
.free = tangox_irq_mux_domain_free,
.translate = tangox_irq_mux_domain_translate,
};

static int __init tangox_irq_mux_init(struct device_node *node,
struct tangox_irq_mux *irq_mux_context)
{
int i, per_irq_reg_size, max_in, max_out, offset, entry, entry_size;
const __be32 *reserved_irq_list;
int ret = -ENOMEM;

DBGLOG("init\n");

if (!irq_mux_context)
return -ENOMEM;

irq_mux_context->reg_base = of_iomap(node, 0);
if (!irq_mux_context->reg_base)
goto err_exit;

irq_mux_context->mux_inputs = IRQ_MUX_INPUT_LINES;
irq_mux_context->mux_outputs = IRQ_MUX_OUTPUT_LINES;

max_in = irq_mux_context->mux_inputs;
max_out = irq_mux_context->mux_outputs;

irq_mux_context->irq_map = kcalloc(max_out, sizeof(int), GFP_KERNEL);
if (!irq_mux_context->irq_map)
goto err_unmap_base;

for (i = 0; i < max_out; i++)
irq_mux_context->irq_map[i] = IRQ_FREE;

// mark reserved IRQ lines
reserved_irq_list = of_get_property(node, "irqs-reserved", &entry_size);
if (reserved_irq_list) {
entry_size /= sizeof(__be32);

DBGLOG("setting up %d reserved irqs\n", entry_size);

for (i = 0; i < entry_size; i++) {
of_property_read_u32_index(node,
"irqs-reserved",
i, &entry);
if (entry >= max_out) {
pr_err("Invalid reserved entry %d > %d: ignored\n", entry, max_out);
continue;
}
irq_mux_context->irq_map[entry] = IRQ_RESERVED;
}
}

DBGLOG("disabling free IRQs\n");

// disable free IRQs during initialisation
for (i = 0; i < max_in; i++) {
tangox_setup_irq_route(irq_mux_context, i, 0);
}

DBGLOG("init backward compatible map\n");

tangox_setup_irq_route(irq_mux_context, 125, 2);
tangox_setup_irq_route(irq_mux_context, 126, 3);
tangox_setup_irq_route(irq_mux_context, 127, 4);

raw_spin_lock_init(&irq_mux_context->lock);

return 0;

err_free_map:
kfree(irq_mux_context->irq_map);
err_unmap_base:
iounmap(irq_mux_context->reg_base);
err_exit:
return ret;
}

static int __init tangox_irq_mux_deinit(struct tangox_irq_mux *irq_mux_context)
{
if (!irq_mux_context)
return -ENOMEM;

if (irq_mux_context->reg_base)
iounmap(irq_mux_context->reg_base);

if (irq_mux_context->irq_map)
kfree(irq_mux_context->irq_map);

kfree(irq_mux_context);

return 0;
}

static int __init tangox_of_irq_mux_init(struct device_node *node,
struct device_node *parent)
{
struct tangox_irq_mux *irq_mux_context;
struct irq_domain *parent_domain, *domain;
int err;

DBGLOG("irqv2 begin\n");

if (!parent) {
pr_err("%s: no parent, giving up\n", node->full_name);
return -ENODEV;
}

parent_domain = irq_find_host(parent);
if (!parent_domain) {
pr_err("%s: unable to obtain parent domain\n", node->full_name);
return -ENXIO;
}

irq_mux_context = kzalloc(sizeof(*irq_mux_context), GFP_KERNEL);

err = tangox_irq_mux_init(node, irq_mux_context);
if (err) {
pr_err("%s: init failed (%d)\n", node->full_name, err);
tangox_irq_mux_deinit(irq_mux_context);
return err;
}

domain = irq_domain_add_hierarchy(parent_domain, 0,
irq_mux_context->mux_inputs,
node, &tangox_irq_mux_domain_ops,
irq_mux_context);
if (!domain) {
pr_err("%s: failed to allocated domain\n", node->full_name);
tangox_irq_mux_deinit(irq_mux_context);
return -ENOMEM;
}

return 0;

}
IRQCHIP_DECLARE(tangox_intc, "sigma,smp-irq-mux", tangox_of_irq_mux_init);