Re: [PATCH v6 3/6] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

From: Vivek Gautam
Date: Thu Feb 01 2018 - 07:32:10 EST



On 2/1/2018 5:03 PM, Sricharan R wrote:
Hi Robin,

On 1/31/2018 6:36 PM, Robin Murphy wrote:
On 19/01/18 11:43, Vivek Gautam wrote:
From: Sricharan R <sricharan@xxxxxxxxxxxxxx>

The smmu device probe/remove and add/remove master device callbacks
gets called when the smmu is not linked to its master, that is without
the context of the master device. So calling runtime apis in those places
separately.

Signed-off-by: Sricharan R <sricharan@xxxxxxxxxxxxxx>
[vivek: Cleanup pm runtime calls]
Signed-off-by: Vivek Gautam <vivek.gautam@xxxxxxxxxxxxxx>
---
 drivers/iommu/arm-smmu.c | 45 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 41 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 21acffe91a1c..95478bfb182c 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -914,11 +914,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
ÂÂÂÂÂ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
ÂÂÂÂÂ struct arm_smmu_device *smmu = smmu_domain->smmu;
ÂÂÂÂÂ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
-ÂÂÂ int irq;
+ÂÂÂ int ret, irq;
 Â if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
ÂÂÂÂÂÂÂÂÂ return;
 + ret = pm_runtime_get_sync(smmu->dev);
+ÂÂÂ if (ret)
+ÂÂÂÂÂÂÂ return;
+
ÂÂÂÂÂ /*
ÂÂÂÂÂÂ * Disable the context bank and free the page tables before freeing
ÂÂÂÂÂÂ * it.
@@ -933,6 +937,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
 Â free_io_pgtable_ops(smmu_domain->pgtbl_ops);
ÂÂÂÂÂ __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
+
+ÂÂÂ pm_runtime_put_sync(smmu->dev);
 }
  static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
@@ -1408,12 +1414,20 @@ static int arm_smmu_add_device(struct device *dev)
ÂÂÂÂÂ while (i--)
ÂÂÂÂÂÂÂÂÂ cfg->smendx[i] = INVALID_SMENDX;
 - ret = arm_smmu_master_alloc_smes(dev);
+ÂÂÂ ret = pm_runtime_get_sync(smmu->dev);
ÂÂÂÂÂ if (ret)
ÂÂÂÂÂÂÂÂÂ goto out_cfg_free;
 + ret = arm_smmu_master_alloc_smes(dev);
+ÂÂÂ if (ret) {
+ÂÂÂÂÂÂÂ pm_runtime_put_sync(smmu->dev);
+ÂÂÂÂÂÂÂ goto out_cfg_free;
Please keep to the existing pattern and put this on the cleanup path with a new label, rather than inline.
ok.

+ÂÂÂ }
+
ÂÂÂÂÂ iommu_device_link(&smmu->iommu, dev);
 + pm_runtime_put_sync(smmu->dev);
+
ÂÂÂÂÂ return 0;
  out_cfg_free:
@@ -1428,7 +1442,7 @@ static void arm_smmu_remove_device(struct device *dev)
ÂÂÂÂÂ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
ÂÂÂÂÂ struct arm_smmu_master_cfg *cfg;
ÂÂÂÂÂ struct arm_smmu_device *smmu;
-
+ÂÂÂ int ret;
 Â if (!fwspec || fwspec->ops != &arm_smmu_ops)
ÂÂÂÂÂÂÂÂÂ return;
@@ -1436,8 +1450,21 @@ static void arm_smmu_remove_device(struct device *dev)
 cfg = fwspec->iommu_priv;
ÂÂÂÂÂ smmu = cfg->smmu;
 + /*
+ÂÂÂÂ * The device link between the master device and
+ÂÂÂÂ * smmu is already purged at this point.
+ÂÂÂÂ * So enable the power to smmu explicitly.
+ÂÂÂÂ */
I don't understand this comment, especially since we don't even introduce device links until the following patch... :/

This is because the core device_del callback, does a device_links_purge for that device,
before calling the remove_device notifier. As a result, have to explicitly turn on the
power to iommu. Probably the comment should be removed, rest of the places we don't
explain why we are turning on explicitly.

Yes, will remove the comment here.


+
+ÂÂÂ ret = pm_runtime_get_sync(smmu->dev);
+ÂÂÂ if (ret)
+ÂÂÂÂÂÂÂ return;
+
ÂÂÂÂÂ iommu_device_unlink(&smmu->iommu, dev);
ÂÂÂÂÂ arm_smmu_master_free_smes(fwspec);
+
+ÂÂÂ pm_runtime_put_sync(smmu->dev);
+
ÂÂÂÂÂ iommu_group_remove_device(dev);
ÂÂÂÂÂ kfree(fwspec->iommu_priv);
ÂÂÂÂÂ iommu_fwspec_free(dev);
@@ -2130,6 +2157,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
ÂÂÂÂÂ if (err)
ÂÂÂÂÂÂÂÂÂ return err;
 + platform_set_drvdata(pdev, smmu);
+
+ÂÂÂ pm_runtime_enable(dev);
+
+ÂÂÂ err = pm_runtime_get_sync(dev);
+ÂÂÂ if (err)
+ÂÂÂÂÂÂÂ return err;
+
ÂÂÂÂÂ err = arm_smmu_device_cfg_probe(smmu);
ÂÂÂÂÂ if (err)
ÂÂÂÂÂÂÂÂÂ return err;
@@ -2171,9 +2206,9 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
ÂÂÂÂÂÂÂÂÂ return err;
ÂÂÂÂÂ }
 - platform_set_drvdata(pdev, smmu);
ÂÂÂÂÂ arm_smmu_device_reset(smmu);
ÂÂÂÂÂ arm_smmu_test_smr_masks(smmu);
+ÂÂÂ pm_runtime_put_sync(dev);
 Â /*
ÂÂÂÂÂÂ * For ACPI and generic DT bindings, an SMMU will be probed before
@@ -2212,6 +2247,8 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 Â /* Turn the thing off */
ÂÂÂÂÂ writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
+ÂÂÂ pm_runtime_force_suspend(smmu->dev);
Why do we need this? I guess it might be a Qualcomm-ism as I don't see anyone else calling it from .remove other than a couple of other qcom_* drivers. Given that we only get here during system shutdown (or the root user intentionally pissing about with driver unbinding), it doesn't seem like a point where power saving really matters all that much.

I'd also naively expect that anything this device was the last consumer off would get turned off by core code anyway once it's removed, but maybe things aren't that slick; I dunno :/
hmm, that should not be needed. with turning of all consumers taken care by device_link code before
the supplier (iommu) remove gets called should ensure that. So the above force_suspend should
not be needed/can be removed. But one more thing is, we do touch the register in the above code.
So that should require a additional get/put sync around that writel.

Possibly we can replace the force_suspend() with a pm_runtime_disable() to complement pm_runtime_enable in the probe.
I will test the scenario where we are writing the SMMU register in .remove path.

regards
Vivek


Regards,
Sricharan