Re: [PATCH] iommu/vt-d: Remove unnecessary locking in intel_irq_remapping_alloc()

[Baolu Lu]

On 3/14/23 11:54 PM, Jacob Pan wrote:
> Hi BaoLu,
>
> On Tue, 14 Mar 2023 13:18:36 +0800, Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx>
> wrote:
>
> > The global rwsem dmar_global_lock was introduced by commit 3a5670e8ac932
> > ("iommu/vt-d: Introduce a rwsem to protect global data structures"). It
> > is used to protect DMAR related global data from DMAR hotplug operations.
> >
> > Using dmar_global_lock in intel_irq_remapping_alloc() is unnecessary as
> > the DMAR global data structures are not touched there. Remove it to avoid
> > below lockdep warning.
> >
> >   ======================================================
> >   WARNING: possible circular locking dependency detected
> >   6.3.0-rc2 #468 Not tainted
> >   ------------------------------------------------------
> >   swapper/0/1 is trying to acquire lock:
> >   ff1db4cb40178698 (&domain->mutex){+.+.}-{3:3},
> >     at: __irq_domain_alloc_irqs+0x3b/0xa0
> >
> >   but task is already holding lock:
> >   ffffffffa0c1cdf0 (dmar_global_lock){++++}-{3:3},
> >     at: intel_iommu_init+0x58e/0x880
> >
> >   which lock already depends on the new lock.
> >
> >   the existing dependency chain (in reverse order) is:
> >
> >   -> #1 (dmar_global_lock){++++}-{3:3}:
> >          lock_acquire+0xd6/0x320
> >          down_read+0x42/0x180
> >          intel_irq_remapping_alloc+0xad/0x750
> >          mp_irqdomain_alloc+0xb8/0x2b0
> >          irq_domain_alloc_irqs_locked+0x12f/0x2d0
> >          __irq_domain_alloc_irqs+0x56/0xa0
> >          alloc_isa_irq_from_domain.isra.7+0xa0/0xe0
> >          mp_map_pin_to_irq+0x1dc/0x330
> >          setup_IO_APIC+0x128/0x210
> >          apic_intr_mode_init+0x67/0x110
> >          x86_late_time_init+0x24/0x40
> >          start_kernel+0x41e/0x7e0
> >          secondary_startup_64_no_verify+0xe0/0xeb
> >
> >   -> #0 (&domain->mutex){+.+.}-{3:3}:
> >          check_prevs_add+0x160/0xef0
> >          __lock_acquire+0x147d/0x1950
> >          lock_acquire+0xd6/0x320
> >          __mutex_lock+0x9c/0xfc0
> >          __irq_domain_alloc_irqs+0x3b/0xa0
> >          dmar_alloc_hwirq+0x9e/0x120
> >          iommu_pmu_register+0x11d/0x200
> >          intel_iommu_init+0x5de/0x880
> >          pci_iommu_init+0x12/0x40
> >          do_one_initcall+0x65/0x350
> >          kernel_init_freeable+0x3ca/0x610
> >          kernel_init+0x1a/0x140
> >          ret_from_fork+0x29/0x50
> >
> >   other info that might help us debug this:
> >
> >   Possible unsafe locking scenario:
> >
> >          CPU0                    CPU1
> >          ----                    ----
> >     lock(dmar_global_lock);
> >                                  lock(&domain->mutex);
> >                                  lock(dmar_global_lock);
> >     lock(&domain->mutex);
> >
> >                  *** DEADLOCK ***
> >
> > Fixes: 9dbb8e3452ab ("irqdomain: Switch to per-domain locking")
> > Signed-off-by: Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx>
> > ---
> >   drivers/iommu/intel/irq_remapping.c | 6 ------
> >   1 file changed, 6 deletions(-)
> >
> > diff --git a/drivers/iommu/intel/irq_remapping.c
> > b/drivers/iommu/intel/irq_remapping.c index 6d01fa078c36..df9e261af0b5
> > 100644 --- a/drivers/iommu/intel/irq_remapping.c
> > +++ b/drivers/iommu/intel/irq_remapping.c
> > @@ -311,14 +311,12 @@ static int set_ioapic_sid(struct irte *irte, int
> > apic) if (!irte)
> >   		return -1;
> >   
> > -	down_read(&dmar_global_lock);
> >   	for (i = 0; i < MAX_IO_APICS; i++) {
> >   		if (ir_ioapic[i].iommu && ir_ioapic[i].id == apic) {
> >   			sid = (ir_ioapic[i].bus << 8) |
> > ir_ioapic[i].devfn; break;
> >   		}
> >   	}
> > -	up_read(&dmar_global_lock);
> >   
> >   	if (sid == 0) {
> >   		pr_warn("Failed to set source-id of IOAPIC (%d)\n",
> > apic); @@ -338,14 +336,12 @@ static int set_hpet_sid(struct irte *irte,
> > u8 id) if (!irte)
> >   		return -1;
> >   
> > -	down_read(&dmar_global_lock);
> >   	for (i = 0; i < MAX_HPET_TBS; i++) {
> >   		if (ir_hpet[i].iommu && ir_hpet[i].id == id) {
> >   			sid = (ir_hpet[i].bus << 8) | ir_hpet[i].devfn;
> >   			break;
> >   		}
> >   	}
> > -	up_read(&dmar_global_lock);
> >   
> >   	if (sid == 0) {
> >   		pr_warn("Failed to set source-id of HPET block (%d)\n",
> > id); @@ -1339,9 +1335,7 @@ static int intel_irq_remapping_alloc(struct
> > irq_domain *domain, if (!data)
> >   		goto out_free_parent;
> >   
> > -	down_read(&dmar_global_lock);
> >   	index = alloc_irte(iommu, &data->irq_2_iommu, nr_irqs);
> > -	up_read(&dmar_global_lock);
> >   	if (index < 0) {
> >   		pr_warn("Failed to allocate IRTE\n");
> >   		kfree(data);
> Reviewed-by: Jacob Pan <jacob.jun.pan@xxxxxxxxxxxxxxx>
>
> slightly beyond the scope of this, do we need to take dmar_global_lock
> below? shouldn't it be in single threaded context?
>
> 	down_write(&dmar_global_lock);
> 	ret = dmar_dev_scope_init();
> 	up_write(&dmar_global_lock);
>
> 	return ret;
> }
> rootfs_initcall(ir_dev_scope_init);

Yes, agreed. This runs in a single threaded context. I will remove the
locking in a cleanup patch.

Best regards,
baolu