Re: [PATCH 1/1] iommu/vt-d: Prevent boot failure with devices requiring ATS
From: Baolu Lu
Date: Wed Sep 04 2024 - 22:53:59 EST
On 9/4/24 4:17 PM, Tian, Kevin wrote:
From: Baolu Lu <baolu.lu@xxxxxxxxxxxxxxx>
Sent: Wednesday, September 4, 2024 3:50 PM
On 2024/9/4 14:49, Tian, Kevin wrote:
From: Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx>
Sent: Wednesday, September 4, 2024 2:07 PM
SOC-integrated devices on some platforms require their PCI ATS enabled
for operation when the IOMMU is in scalable mode. Those devices are
reported via ACPI/SATC table with the ATC_REQUIRED bit set in the Flags
field.
The PCI subsystem offers the 'pci=noats' kernel command to disable PCI
ATS on all devices. Using 'pci=noat' with devices that require PCI ATS
can cause a conflict, leading to boot failure, especially if the device
is a graphics device.
To prevent this issue, check PCI ATS support before enumerating the
IOMMU
devices. If any device requires PCI ATS, but PCI ATS is disabled by
'pci=noats', switch the IOMMU to operate in legacy mode to ensure
successful booting.
I guess the reason of switching to legacy mode is because the platform
automatically enables ATS in this mode, as the comment says in
dmar_ats_supported(). This should be explained otherwise it's unclear
why switching the mode can make ATS working for those devices.
Not 'automatically enable ATS,' but hardware provides something that is
equivalent to PCI ATS. The ATS capability on the device is still
disabled. That's the reason why such device must be an SOC-integrated
one.
well does that equivalent means use PCI ATS protocol at all (i.e. do
untranslated request followed by translated request based on device
TLB)?
If yes it's still ATS under the hood.
If not could you elaborate how it works in PCI world?
I'm not a hardware expert, so I can't provide specific details. :-)
Anyway, from the Linux box's perspective, if 'pci=noats' is used on a
Meteorlake device, the 'lspci' tool shows that PCI ATS is disabled:
# dmesg
[...]
[ 2.419806] pci 0000:00:02.0: DMAR: PCI/ATS not supported, system
working in IOMMU legacy mode
[...]
# lspci -s 0000:00:02.0 -vv
00:02.0 VGA compatible controller: Intel Corporation Meteor Lake-M
[Intel Graphics] (prog-if 00 [VGA controller])
[...]
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable-, Smallest Translation Unit: 00
[...]
As for how hardware works, it appears to be transparent to the software.
But then doesn't it break the meaning of 'pci=noats' which means
disabling ATS physically? It's described as "do not use PCIe ATS and
IOMMU device IOTLB" in kernel doc, which is not equivalent to
"leave PCIe ATS to be managed by HW".
Therefore, the PCI ATS is not used and the syntax of pci=noats is not
broken.
I'm not sure the point of noats is to just disable the PCI capability
while allowing the underlying hw to continue sending ATS protocol...
and why would one want to use 'pci=noats' on a platform which
requires ats?
We don't recommend users to disable ATS on a platform which has devices
that rely on it. But nothing can prevent users from doing so. I am not
sure why it is needed. One possible reason that I can think of is about
security. Sometimes, people don't trust ATS because it allows devices to
access the memory with translated requests directly without any
permission check on the IOMMU end.
but this doesn't make sense. If the user doesn't trust ATS and deliberately
wants to disable ats then it should be followed and whatever usage
requiring ATS is then broken. The user should decide which is more
favored between security vs. usage to make the right call.
Yes. That's just a reason that I could think of.
Anyway, for a client platform, we should avoid any boot failure no
matter what kernel parameters the users are using.
Instead, perhaps we could emit a big fat warning to the user, informing
them that a device relies on ATS for functionality, thus 'pci=noats'
might be compromised.
Thanks,
baolu