Re: [PATCH v3 0/2] iommu: handle drivers that manage iommu directly

[Robin Murphy]

On 06/09/2019 22:44, Rob Clark wrote:
> From: Rob Clark <robdclark@xxxxxxxxxxxx>
>
> One of the challenges we have to enable the aarch64 laptops upstream
> is dealing with the fact that the bootloader enables the display and
> takes the corresponding SMMU context-bank out of BYPASS.  Unfortunately,
> currently, the IOMMU framework attaches a DMA (or potentially an
> IDENTITY) domain before the driver is probed and has a chance to
> intervene and shutdown scanout.  Which makes things go horribly wrong.

Nope, things already went horribly wrong in arm_smmu_device_reset() - 
sure, sometimes for some configurations it might *seem* like they didn't 
and that you can fudge the context bank state at arm's length from core 
code later, but the truth is that impl->cfg_probe is your last chance to 
guarantee that any necessary SMMU state is preserved.

The remainder of the problem involves reworking default domain 
allocation such that we can converge on what iommu_request_dm_for_dev() 
currently does but without the momentary attachment to a translation 
domain to cause hiccups. That's starting here:

https://lore.kernel.org/linux-iommu/cover.1566353521.git.sai.praneeth.prakhya@xxxxxxxxx/

> But in this case, drm/msm is already directly managing it's IOMMUs
> directly, the DMA API attached iommu_domain simply gets in the way.
> This series adds a way that a driver can indicate to drivers/iommu
> that it does not wish to have an DMA managed iommu_domain attached.
> This way, drm/msm can shut down scanout cleanly before attaching it's
> own iommu_domain.
>
> NOTE that to get things working with arm-smmu on the aarch64 laptops,
> you also need a patchset[1] from Bjorn Andersson to inherit SMMU config
> at boot, when it is already enabled.
>
> [1] https://www.spinics.net/lists/arm-kernel/msg732246.html
>
> NOTE that in discussion of previous revisions, RMRR came up.  This is
> not really a replacement for RMRR (nor does RMRR really provide any
> more information than we already get from EFI GOP, or DT in the
> simplefb case).  I also don't see how RMRR could help w/ SMMU handover
> of CB/SMR config (Bjorn's patchset[1]) without defining new tables.

The point of RMRR-like-things is that they identify not just the memory 
region but also the specific device accessing them, which means the 
IOMMU driver knows up-front which IDs etc. it must be careful not to 
disrupt. Obviously for SMMU that *would* be some new table (designed to 
encompass everything relevant) since literal RMRRs are specifically an 
Intel VT-d thing.

> This perhaps doesn't solve the more general case of bootloader enabled
> display for drivers that actually want to use DMA API managed IOMMU.
> But it does also happen to avoid a related problem with GPU, caused by
> the DMA domain claiming the context bank that the GPU firmware expects
> to use.

Careful bringing that up again, or I really will rework the context bank 
allocator to avoid this default domain problem entirely... ;)

Robin.

>  And it avoids spurious TLB invalidation coming from the unused
> DMA domain.  So IMHO this is a useful and necessary change.
>
> Rob Clark (2):
>    iommu: add support for drivers that manage iommu explicitly
>    drm/msm: mark devices where iommu is managed by driver
>
>   drivers/gpu/drm/msm/adreno/adreno_device.c | 1 +
>   drivers/gpu/drm/msm/disp/dpu1/dpu_kms.c    | 1 +
>   drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c   | 1 +
>   drivers/gpu/drm/msm/msm_drv.c              | 1 +
>   drivers/iommu/iommu.c                      | 2 +-
>   drivers/iommu/of_iommu.c                   | 3 +++
>   include/linux/device.h                     | 3 ++-
>   7 files changed, 10 insertions(+), 2 deletions(-)