Re: [PATCH 1/2] x86/amd-iommu: enable iommu before attachingdevices

From: Vivek Goyal
Date: Fri Apr 02 2010 - 12:00:40 EST

On Thu, Apr 01, 2010 at 06:23:53PM -0700, Chris Wright wrote:
> Hit another kdump problem as reported by Neil Horman. When initializaing
> the IOMMU, we attach devices to their domains before the IOMMU is
> fully (re)initialized. Attaching a device will issue some important
> invalidations. In the context of the newly kexec'd kdump kernel, the
> IOMMU may have stale cached data from the original kernel. Because we
> do the attach too early, the invalidation commands are placed in the new
> command buffer before the IOMMU is updated w/ that buffer. This leaves
> the stale entries in the kdump context and can renders device unusable.
> Simply enable the IOMMU before we do the attach.
> Cc: Neil Horman <nhorman@xxxxxxxxxxxxx>
> Cc: Vivek Goyal <vgoyal@xxxxxxxxxx>
> Signed-off-by: Chris Wright <chrisw@xxxxxxxxxxxx>
> ---
> arch/x86/kernel/amd_iommu_init.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> --- a/arch/x86/kernel/amd_iommu_init.c
> +++ b/arch/x86/kernel/amd_iommu_init.c
> @@ -1288,6 +1288,8 @@ static int __init amd_iommu_init(void)
> if (ret)
> goto free;
> + enable_iommus();
> +
> if (iommu_pass_through)
> ret = amd_iommu_init_passthrough();
> else
> @@ -1300,8 +1302,6 @@ static int __init amd_iommu_init(void)
> amd_iommu_init_notifier();
> - enable_iommus();
> -

Ok, so now we do enable_iommu() before we attach devices and flush DTE and
domain PDE, IO TLB. That makes sense.

Following is just for my education purposes. Trying to understand better
the impact of in-flight DMAs. So IIUC, in kudmp context following seems to be
sequence of events.

1. kernel crashes, we leave IOMMU enabled.

2. boot into capture kernel and we call enable_iommus(). This function
first disables iommu, sets up new device table and enables iommus
a. So during this small window when iommu is disabled and we enable
it back, any inflight DMA will passthrough possibly to an
unintended physical address as translation is disabled and it
can corrupt the kdump kenrel.

b. Even after enabling the iommu, I guess we will continue to
use cached DTE, and translation information to handle any
in-flight DMA. The difference is that now iommus are enabled
so any in-flight DMA should go to the address as intended in
first kenrel and should not corrupt anything.

3. Once iommus are enabled again, we allocated and initilize protection
domains. We attach devices to domains. In the process we flush the

c. Looks like do_attach->set_dte_entry(), by default gives write
permission (IW) to all the devices. I am assuming that at
this point of time translation is enabled and possibly unity
mapped. If that's the case, any in-flight DMA will not be
allowed to happen at unity mapped address and this can possibly
corrupt the kernel?

I understand this patch should fix the case when in second kernel a
device is not doing DMA because of possibly cached DTE, and translation
information. But looks like in-flight DMA issues will still need a closer
look. But that is a separate issue and needs to be addressed in separate
set of patches.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at