Re: [PATCH] ARM: KVM: add irqfd and irq routing support

[Anup Patel]

On Fri, May 9, 2014 at 6:15 PM, Eric Auger <eric.auger@xxxxxxxxxx> wrote:
> This patch enables irqfd and irq routing on ARM.
>
> It turns on CONFIG_HAVE_KVM_EVENTFD and CONFIG_HAVE_KVM_IRQ_ROUTING
>
> irqfd framework enables to assign physical IRQs to guests.
>
> 1) user-side uses KVM_IRQFD VM ioctl to pass KVM a kvm_irqfd struct that
> associates a VM, an eventfd, an IRQ number (aka. the GSI). When an actor
> signals the eventfd (typically a VFIO platform driver), the irqfd subsystem
> injects the specified IRQ into the VM (the "GSI" takes the semantic of a
> virtual IRQ for that guest).
>
> 2) the other use case is user-side does 1) and uses KVM_SET_GSI_ROUTING
> VM ioctl to create an association between a VM, a physical IRQ (aka GSI) and
> a virtual IRQ (aka irchip.pin).  This creates a so-called GSI routing entry.
> When someone triggers the eventfd, irqfd handles it but uses the specified
> routing and eventually injects irqchip.pin virtual IRQ into the guest. In that
> context the GSI takes the semantic of a physical IRQ while the irqchip.pin
> takes the semantic of a virtual IRQ.
>
> in 1) routing is used by irqfd but an identity routing is created by default
> making the gsi = irqchip.pin. Note on ARM there is a single interrupt
> controller kind, the GIC.
>
> GSI routing mostly is implemented in generic irqchip.c.
> The tiny ARM specific part is directly implemented in the virtual interrupt
> controller (vgic.c) as it is done for powerpc for instance. This option was
> prefered compared to implementing other #ifdef in irq_comm.c (x86 and ia64).
> Hence irq_comm.c is not used at all.
>
> Routing currently is not used for anything else than irqfd IRQ injection. Only
> SPI can be injected. This means the vgic is not totally hidden behind the
> irqchip. There are separate discussions on PPI/SGI routing.
>
> Only level sensitive IRQs are supported (with a registered resampler). As a
> reminder the resampler is a second eventfd called by irqfd framework when the
> virtual IRQ is completed by the guest. This eventfd is supposed to be handled
> on user-side
>
> MSI routing is not supported yet.
>
> This work was tested with Calxeda Midway xgmac main interrupt (with and without
> explicit user routing) with qemu-system-arm and QEMU VFIO platform device.

I have a doubt here:
"Why are we not using EOImode here?"
OR
"Is de-prioritization of routed IRQs taken care by different patch?"

As-per GICv2 spec, ARM recommends using GICC_CTRL.EOImodeNS=1 for
routing physical IRQs as virtual IRQs. Xen already uses it for both arm32 and
arm64. The EOImode is recommended here because we need to de-prioritize
routed IRQs before forwarding them to Guest as virtual IRQs. Later, when
Guests is done processing virtual IRQ it will do EOI after which we can
de-activate the physical IRQ using GICC_DIR register.

Regards,
Anup

>
> Signed-off-by: Eric Auger <eric.auger@xxxxxxxxxx>
>
> Conflicts:
>         Documentation/virtual/kvm/api.txt
>         arch/arm/kvm/Kconfig
> ---
>  Documentation/virtual/kvm/api.txt |   4 +-
>  arch/arm/include/uapi/asm/kvm.h   |   8 +++
>  arch/arm/kvm/Kconfig              |   2 +
>  arch/arm/kvm/Makefile             |   1 +
>  arch/arm/kvm/irq.h                |  25 ++++++++
>  virt/kvm/arm/vgic.c               | 124 ++++++++++++++++++++++++++++++++++++--
>  6 files changed, 158 insertions(+), 6 deletions(-)
>  create mode 100644 arch/arm/kvm/irq.h
>
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index a9380ba5..202d63e3 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1339,7 +1339,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>  4.52 KVM_SET_GSI_ROUTING
>
>  Capability: KVM_CAP_IRQ_ROUTING
> -Architectures: x86 ia64 s390
> +Architectures: x86 ia64 s390 arm
>  Type: vm ioctl
>  Parameters: struct kvm_irq_routing (in)
>  Returns: 0 on success, -1 on error
> @@ -2126,7 +2126,7 @@ into the hash PTE second double word).
>  4.75 KVM_IRQFD
>
>  Capability: KVM_CAP_IRQFD
> -Architectures: x86
> +Architectures: x86 arm
>  Type: vm ioctl
>  Parameters: struct kvm_irqfd (in)
>  Returns: 0 on success, -1 on error
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index ef0c878..89b864d 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -192,6 +192,14 @@ struct kvm_arch_memory_slot {
>  /* Highest supported SPI, from VGIC_NR_IRQS */
>  #define KVM_ARM_IRQ_GIC_MAX            127
>
> +/* needed by IRQ routing */
> +
> +/* One single KVM irqchip, ie. the VGIC */
> +#define KVM_NR_IRQCHIPS          1
> +
> +/* virtual interrupt controller input pins (max 480 SPI, 32 SGI/PPI) */
> +#define KVM_IRQCHIP_NUM_PINS     256
> +
>  /* PSCI interface */
>  #define KVM_PSCI_FN_BASE               0x95c1ba5e
>  #define KVM_PSCI_FN(n)                 (KVM_PSCI_FN_BASE + (n))
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 4be5bb1..096692c 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -24,6 +24,7 @@ config KVM
>         select KVM_MMIO
>         select KVM_ARM_HOST
>         depends on ARM_VIRT_EXT && ARM_LPAE && !CPU_BIG_ENDIAN
> +       select HAVE_KVM_EVENTFD
>         ---help---
>           Support hosting virtualized guest machines. You will also
>           need to select one or more of the processor modules below.
> @@ -56,6 +57,7 @@ config KVM_ARM_VGIC
>         bool "KVM support for Virtual GIC"
>         depends on KVM_ARM_HOST && OF
>         select HAVE_KVM_IRQCHIP
> +       select HAVE_KVM_IRQ_ROUTING
>         default y
>         ---help---
>           Adds support for a hardware assisted, in-kernel GIC emulation.
> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> index 789bca9..29de111 100644
> --- a/arch/arm/kvm/Makefile
> +++ b/arch/arm/kvm/Makefile
> @@ -21,4 +21,5 @@ obj-y += kvm-arm.o init.o interrupts.o
>  obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
>  obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o
>  obj-$(CONFIG_KVM_ARM_VGIC) += $(KVM)/arm/vgic.o
> +obj-$(CONFIG_HAVE_KVM_EVENTFD) += $(KVM)/eventfd.o $(KVM)/irqchip.o
>  obj-$(CONFIG_KVM_ARM_TIMER) += $(KVM)/arm/arch_timer.o
> diff --git a/arch/arm/kvm/irq.h b/arch/arm/kvm/irq.h
> new file mode 100644
> index 0000000..4d6fcc6
> --- /dev/null
> +++ b/arch/arm/kvm/irq.h
> @@ -0,0 +1,25 @@
> +/*
> + * Copyright (C) 2014, STMicroelectronics
> + * Authors: Eric Auger <eric.auger@xxxxxx>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + */
> +
> +#ifndef __IRQ_H
> +#define __IRQ_H
> +
> +#include <linux/kvm_host.h>
> +/*
> + * Placeholder for irqchip and irq/msi routing declarations
> + * included in irqchip.c
> + */
> +
> +#endif
> diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
> index 56ff9be..a041c29 100644
> --- a/virt/kvm/arm/vgic.c
> +++ b/virt/kvm/arm/vgic.c
> @@ -93,6 +93,9 @@ static struct device_node *vgic_node;
>  #define ACCESS_WRITE_VALUE     (3 << 1)
>  #define ACCESS_WRITE_MASK(x)   ((x) & (3 << 1))
>
> +static struct kvm_irq_routing_entry identity_table[VGIC_NR_IRQS];
> +static int set_default_routing_table(struct kvm *kvm);
> +
>  static void vgic_retire_disabled_irqs(struct kvm_vcpu *vcpu);
>  static void vgic_update_state(struct kvm *kvm);
>  static void vgic_kick_vcpus(struct kvm *kvm);
> @@ -1172,6 +1175,8 @@ static bool vgic_process_maintenance(struct kvm_vcpu *vcpu)
>  {
>         struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>         bool level_pending = false;
> +       struct kvm *kvm;
> +       int is_assigned_irq;
>
>         kvm_debug("MISR = %08x\n", vgic_cpu->vgic_misr);
>
> @@ -1189,12 +1194,22 @@ static bool vgic_process_maintenance(struct kvm_vcpu *vcpu)
>                         vgic_irq_clear_active(vcpu, irq);
>                         vgic_cpu->vgic_lr[lr] &= ~GICH_LR_EOI;
>
> +                       kvm = vcpu->kvm;
> +                       is_assigned_irq = kvm_irq_has_notifier(kvm, 0, irq);
> +
>                         /* Any additional pending interrupt? */
> -                       if (vgic_dist_irq_is_pending(vcpu, irq)) {
> -                               vgic_cpu_irq_set(vcpu, irq);
> -                               level_pending = true;
> -                       } else {
> +                       if (is_assigned_irq) {
>                                 vgic_cpu_irq_clear(vcpu, irq);
> +                               kvm_debug("EOI irqchip routed vIRQ %d\n", irq);
> +                               kvm_notify_acked_irq(kvm, 0, irq);
> +                               vgic_dist_irq_clear(vcpu, irq);
> +                       } else {
> +                               if (vgic_dist_irq_is_pending(vcpu, irq)) {
> +                                       vgic_cpu_irq_set(vcpu, irq);
> +                                       level_pending = true;
> +                               } else {
> +                                       vgic_cpu_irq_clear(vcpu, irq);
> +                               }
>                         }
>
>                         /*
> @@ -1627,6 +1642,8 @@ int kvm_vgic_create(struct kvm *kvm)
>         kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>         kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>
> +       set_default_routing_table(kvm);
> +
>  out_unlock:
>         for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>                 vcpu = kvm_get_vcpu(kvm, vcpu_lock_idx);
> @@ -2017,3 +2034,102 @@ struct kvm_device_ops kvm_arm_vgic_v2_ops = {
>         .get_attr = vgic_get_attr,
>         .has_attr = vgic_has_attr,
>  };
> +
> +
> +/*
> + * set up a default identity routing table
> + * The user-side can further change the routing table using
> + * KVM_SET_GSI_ROUTING VM ioctl
> + */
> +
> +static int set_default_routing_table(struct kvm *kvm)
> +{
> +       struct kvm_irq_routing_entry;
> +       int i;
> +       for (i = 0; i < VGIC_NR_IRQS; i++)      {
> +               identity_table[i].gsi = i;
> +               identity_table[i].type = KVM_IRQ_ROUTING_IRQCHIP;
> +               identity_table[i].u.irqchip.irqchip = 0;
> +               identity_table[i].u.irqchip.pin = i;
> +       }
> +       return kvm_set_irq_routing(kvm, identity_table,
> +                                  ARRAY_SIZE(identity_table), 0);
> +}
> +
> +
> +/*
> + * Functions needed for GSI routing (used by irqchip.c)
> + * implemented in irq_comm.c for x86 and ia64
> + * in architecture specific files for some other archictures (powerpc)
> + */
> +
> +static int vgic_set_assigned_irq(struct kvm_kernel_irq_routing_entry *e,
> +                          struct kvm *kvm, int irq_source_id, int level,
> +                          bool line_status)
> +{
> +       /* only SPI can be assigned in current implementation */
> +       if (e->irqchip.pin < 32)
> +               return -EINVAL;
> +
> +       if (irq_source_id == KVM_USERSPACE_IRQ_SOURCE_ID) {
> +               /*
> +                * This path is not tested yet,
> +                * only irqchip with resampler was exercised
> +                */
> +               kvm_vgic_inject_irq(kvm, 0, e->irqchip.pin, level);
> +       } else if (irq_source_id == KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID) {
> +               if (level == 1) {
> +                       kvm_debug("Inject irqchip routed vIRQ %d\n",
> +                                       e->irqchip.pin);
> +                       kvm_vgic_inject_irq(kvm, 0, e->irqchip.pin, level);
> +                       /*
> +                        * toggling down vIRQ wire is directly handled in
> +                        * process_maintenance for this reason:
> +                        * irqfd_resampler_ack is called in
> +                        * process_maintenance which holds the dist lock.
> +                        * irqfd_resampler_ack calls kvm_set_irq
> +                        * which ends_up calling kvm_vgic_inject_irq.
> +                        * This later attempts to take the lock -> deadlock!
> +                        */
> +               }
> +       }
> +       return 0;
> +
> +}
> +
> +/* void implementation requested to compile irqchip.c */
> +
> +int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
> +               struct kvm *kvm, int irq_source_id, int level, bool line_status)
> +{
> +       return 0;
> +}
> +
> +int kvm_set_routing_entry(struct kvm_irq_routing_table *rt,
> +                         struct kvm_kernel_irq_routing_entry *e,
> +                         const struct kvm_irq_routing_entry *ue)
> +{
> +       int r = -EINVAL;
> +
> +       switch (ue->type) {
> +       case KVM_IRQ_ROUTING_IRQCHIP:
> +               e->set = vgic_set_assigned_irq;
> +               e->irqchip.irqchip = ue->u.irqchip.irqchip;
> +               e->irqchip.pin = ue->u.irqchip.pin;
> +               if (e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS)
> +                       goto out;
> +               /* chip[0][virtualID] = physicalID  */
> +               rt->chip[ue->u.irqchip.irqchip][e->irqchip.pin] = ue->gsi;
> +               break;
> +       default:
> +               goto out;
> +       }
> +
> +       r = 0;
> +out:
> +       return r;
> +}
> +
> +
> +
> +
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/