Re: [PATCH v6 3/6] iommu/arm-smmu: Invoke pm_runtime during probe, add/remove device

[Sricharan R]

Hi Robin,

On 1/31/2018 6:36 PM, Robin Murphy wrote:
> On 19/01/18 11:43, Vivek Gautam wrote:
>> From: Sricharan R <sricharan@xxxxxxxxxxxxxx>
>>
>> The smmu device probe/remove and add/remove master device callbacks
>> gets called when the smmu is not linked to its master, that is without
>> the context of the master device. So calling runtime apis in those places
>> separately.
>>
>> Signed-off-by: Sricharan R <sricharan@xxxxxxxxxxxxxx>
>> [vivek: Cleanup pm runtime calls]
>> Signed-off-by: Vivek Gautam <vivek.gautam@xxxxxxxxxxxxxx>
>> ---
>> Â drivers/iommu/arm-smmu.c | 45 +++++++++++++++++++++++++++++++++++++++++----
>> Â 1 file changed, 41 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
>> index 21acffe91a1c..95478bfb182c 100644
>> --- a/drivers/iommu/arm-smmu.c
>> +++ b/drivers/iommu/arm-smmu.c
>> @@ -914,11 +914,15 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>> ÂÂÂÂÂ struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
>> ÂÂÂÂÂ struct arm_smmu_device *smmu = smmu_domain->smmu;
>> ÂÂÂÂÂ struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>> -ÂÂÂ int irq;
>> +ÂÂÂ int ret, irq;
>> Â ÂÂÂÂÂ if (!smmu || domain->type == IOMMU_DOMAIN_IDENTITY)
>> ÂÂÂÂÂÂÂÂÂ return;
>> Â +ÂÂÂ ret = pm_runtime_get_sync(smmu->dev);
>> +ÂÂÂ if (ret)
>> +ÂÂÂÂÂÂÂ return;
>> +
>> ÂÂÂÂÂ /*
>> ÂÂÂÂÂÂ * Disable the context bank and free the page tables before freeing
>> ÂÂÂÂÂÂ * it.
>> @@ -933,6 +937,8 @@ static void arm_smmu_destroy_domain_context(struct iommu_domain *domain)
>> Â ÂÂÂÂÂ free_io_pgtable_ops(smmu_domain->pgtbl_ops);
>> ÂÂÂÂÂ __arm_smmu_free_bitmap(smmu->context_map, cfg->cbndx);
>> +
>> +ÂÂÂ pm_runtime_put_sync(smmu->dev);
>> Â }
>> Â Â static struct iommu_domain *arm_smmu_domain_alloc(unsigned type)
>> @@ -1408,12 +1414,20 @@ static int arm_smmu_add_device(struct device *dev)
>> ÂÂÂÂÂ while (i--)
>> ÂÂÂÂÂÂÂÂÂ cfg->smendx[i] = INVALID_SMENDX;
>> Â -ÂÂÂ ret = arm_smmu_master_alloc_smes(dev);
>> +ÂÂÂ ret = pm_runtime_get_sync(smmu->dev);
>> ÂÂÂÂÂ if (ret)
>> ÂÂÂÂÂÂÂÂÂ goto out_cfg_free;
>> Â +ÂÂÂ ret = arm_smmu_master_alloc_smes(dev);
>> +ÂÂÂ if (ret) {
>> +ÂÂÂÂÂÂÂ pm_runtime_put_sync(smmu->dev);
>> +ÂÂÂÂÂÂÂ goto out_cfg_free;
> 
> Please keep to the existing pattern and put this on the cleanup path with a new label, rather than inline.

 ok.

> 
>> +ÂÂÂ }
>> +
>> ÂÂÂÂÂ iommu_device_link(&smmu->iommu, dev);
>> Â +ÂÂÂ pm_runtime_put_sync(smmu->dev);
>> +
>> ÂÂÂÂÂ return 0;
>> Â Â out_cfg_free:
>> @@ -1428,7 +1442,7 @@ static void arm_smmu_remove_device(struct device *dev)
>> ÂÂÂÂÂ struct iommu_fwspec *fwspec = dev->iommu_fwspec;
>> ÂÂÂÂÂ struct arm_smmu_master_cfg *cfg;
>> ÂÂÂÂÂ struct arm_smmu_device *smmu;
>> -
>> +ÂÂÂ int ret;
>> Â ÂÂÂÂÂ if (!fwspec || fwspec->ops != &arm_smmu_ops)
>> ÂÂÂÂÂÂÂÂÂ return;
>> @@ -1436,8 +1450,21 @@ static void arm_smmu_remove_device(struct device *dev)
>>  cfg = fwspec->iommu_priv;
>> ÂÂÂÂÂ smmu = cfg->smmu;
>> Â +ÂÂÂ /*
>> +ÂÂÂÂ * The device link between the master device and
>> +ÂÂÂÂ * smmu is already purged at this point.
>> +ÂÂÂÂ * So enable the power to smmu explicitly.
>> +ÂÂÂÂ */
> 
> I don't understand this comment, especially since we don't even introduce device links until the following patch... :/
> 

 This is because the core device_del callback, does a device_links_purge for that device,
 before calling the remove_device notifier. As a result, have to explicitly turn on the
 power to iommu. Probably the comment should be removed, rest of the places we don't
 explain why we are turning on explicitly.

>> +
>> +ÂÂÂ ret = pm_runtime_get_sync(smmu->dev);
>> +ÂÂÂ if (ret)
>> +ÂÂÂÂÂÂÂ return;
>> +
>> ÂÂÂÂÂ iommu_device_unlink(&smmu->iommu, dev);
>> ÂÂÂÂÂ arm_smmu_master_free_smes(fwspec);
>> +
>> +ÂÂÂ pm_runtime_put_sync(smmu->dev);
>> +
>> ÂÂÂÂÂ iommu_group_remove_device(dev);
>> ÂÂÂÂÂ kfree(fwspec->iommu_priv);
>> ÂÂÂÂÂ iommu_fwspec_free(dev);
>> @@ -2130,6 +2157,14 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>> ÂÂÂÂÂ if (err)
>> ÂÂÂÂÂÂÂÂÂ return err;
>> Â +ÂÂÂ platform_set_drvdata(pdev, smmu);
>> +
>> +ÂÂÂ pm_runtime_enable(dev);
>> +
>> +ÂÂÂ err = pm_runtime_get_sync(dev);
>> +ÂÂÂ if (err)
>> +ÂÂÂÂÂÂÂ return err;
>> +
>> ÂÂÂÂÂ err = arm_smmu_device_cfg_probe(smmu);
>> ÂÂÂÂÂ if (err)
>> ÂÂÂÂÂÂÂÂÂ return err;
>> @@ -2171,9 +2206,9 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
>> ÂÂÂÂÂÂÂÂÂ return err;
>> ÂÂÂÂÂ }
>> Â -ÂÂÂ platform_set_drvdata(pdev, smmu);
>> ÂÂÂÂÂ arm_smmu_device_reset(smmu);
>> ÂÂÂÂÂ arm_smmu_test_smr_masks(smmu);
>> +ÂÂÂ pm_runtime_put_sync(dev);
>> Â ÂÂÂÂÂ /*
>> ÂÂÂÂÂÂ * For ACPI and generic DT bindings, an SMMU will be probed before
>> @@ -2212,6 +2247,8 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
>> Â ÂÂÂÂÂ /* Turn the thing off */
>> ÂÂÂÂÂ writel(sCR0_CLIENTPD, ARM_SMMU_GR0_NS(smmu) + ARM_SMMU_GR0_sCR0);
>> +ÂÂÂ pm_runtime_force_suspend(smmu->dev);
> 
> Why do we need this? I guess it might be a Qualcomm-ism as I don't see anyone else calling it from .remove other than a couple of other qcom_* drivers. Given that we only get here during system shutdown (or the root user intentionally pissing about with driver unbinding), it doesn't seem like a point where power saving really matters all that much.
> 
> I'd also naively expect that anything this device was the last consumer off would get turned off by core code anyway once it's removed, but maybe things aren't that slick; I dunno :/

 hmm, that should not be needed. with turning of all consumers taken care by device_link code before
 the supplier (iommu) remove gets called should ensure that. So the above force_suspend should
 not be needed/can be removed. But one more thing is, we do touch the register in the above code.
 So that should require a additional get/put sync around that writel.

Regards,
 Sricharan

-- 
"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation