Re: [PATCH v3 00/34] iommu: Move iommu_group setup to IOMMU core code
From: Qian Cai
Date: Tue Jun 30 2020 - 20:40:43 EST
On Wed, Apr 29, 2020 at 03:36:38PM +0200, Joerg Roedel wrote:
> Hi,
>
> here is the third version of this patch-set. Older versions can be found
> here:
>
> v1: https://lore.kernel.org/lkml/20200407183742.4344-1-joro@xxxxxxxxxx/
> (Has some more introductory text)
>
> v2: https://lore.kernel.org/lkml/20200414131542.25608-1-joro@xxxxxxxxxx/
>
> Changes v2 -> v3:
>
> * Rebased v5.7-rc3
>
> * Added a missing iommu_group_put() as reported by Lu Baolu.
>
> * Added a patch to consolidate more initialization work in
> __iommu_probe_device(), fixing a bug where no 'struct
> device_iommu' was allocated in the hotplug path.
>
> There is also a git-branch available with these patches applied:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/joro/linux.git/log/?h=iommu-probe-device-v3
>
> Please review. If there are no objections I plan to put these patches
> into the IOMMU tree early next week.
Looks like this patchset introduced an use-after-free on arm-smmu-v3.
Reproduced using mlx5,
# echo 1 > /sys/class/net/enp11s0f1np1/device/sriov_numvfs
# echo 0 > /sys/class/net/enp11s0f1np1/device/sriov_numvfs
The .config,
https://github.com/cailca/linux-mm/blob/master/arm64.config
Looking at the free stack,
iommu_release_device->iommu_group_remove_device
was introduced in 07/34 ("iommu: Add probe_device() and release_device()
call-backs").
[ 9426.724641][ T3356] pci 0000:0b:01.2: Removing from iommu group 3
[ 9426.731347][ T3356] ==================================================================
[ 9426.739263][ T3356] BUG: KASAN: use-after-free in __lock_acquire+0x3458/0x4440
__lock_acquire at kernel/locking/lockdep.c:4250
[ 9426.746477][ T3356] Read of size 8 at addr ffff0089df1a6f68 by task bash/3356
[ 9426.753601][ T3356]
[ 9426.755782][ T3356] CPU: 5 PID: 3356 Comm: bash Not tainted 5.8.0-rc3-next-20200630 #2
[ 9426.763687][ T3356] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019
[ 9426.774111][ T3356] Call trace:
[ 9426.777245][ T3356] dump_backtrace+0x0/0x398
[ 9426.781593][ T3356] show_stack+0x14/0x20
[ 9426.785596][ T3356] dump_stack+0x140/0x1b8
[ 9426.789772][ T3356] print_address_description.isra.12+0x54/0x4a8
[ 9426.795855][ T3356] kasan_report+0x134/0x1b8
[ 9426.800203][ T3356] __asan_report_load8_noabort+0x2c/0x50
[ 9426.805679][ T3356] __lock_acquire+0x3458/0x4440
[ 9426.810373][ T3356] lock_acquire+0x204/0xf10
[ 9426.814722][ T3356] _raw_spin_lock_irqsave+0xf8/0x180
[ 9426.819853][ T3356] arm_smmu_detach_dev+0xd8/0x4a0
arm_smmu_detach_dev at drivers/iommu/arm-smmu-v3.c:2776
[ 9426.824721][ T3356] arm_smmu_release_device+0xb4/0x1c8
arm_smmu_disable_pasid at drivers/iommu/arm-smmu-v3.c:2754
(inlined by) arm_smmu_release_device at drivers/iommu/arm-smmu-v3.c:3000
[ 9426.829937][ T3356] iommu_release_device+0xc0/0x178
iommu_release_device at drivers/iommu/iommu.c:302
[ 9426.834892][ T3356] iommu_bus_notifier+0x118/0x160
[ 9426.839762][ T3356] notifier_call_chain+0xa4/0x128
[ 9426.844630][ T3356] __blocking_notifier_call_chain+0x70/0xa8
[ 9426.850367][ T3356] blocking_notifier_call_chain+0x14/0x20
[ 9426.855929][ T3356] device_del+0x618/0xa00
[ 9426.860105][ T3356] pci_remove_bus_device+0x108/0x2d8
[ 9426.865233][ T3356] pci_stop_and_remove_bus_device+0x1c/0x28
[ 9426.870972][ T3356] pci_iov_remove_virtfn+0x228/0x368
[ 9426.876100][ T3356] sriov_disable+0x8c/0x348
[ 9426.880447][ T3356] pci_disable_sriov+0x5c/0x70
[ 9426.885117][ T3356] mlx5_core_sriov_configure+0xd8/0x260 [mlx5_core]
[ 9426.891549][ T3356] sriov_numvfs_store+0x240/0x318
[ 9426.896417][ T3356] dev_attr_store+0x38/0x68
[ 9426.900766][ T3356] sysfs_kf_write+0xdc/0x128
[ 9426.905200][ T3356] kernfs_fop_write+0x23c/0x448
[ 9426.909897][ T3356] __vfs_write+0x54/0xe8
[ 9426.913984][ T3356] vfs_write+0x124/0x3f0
[ 9426.918070][ T3356] ksys_write+0xe8/0x1b8
[ 9426.922157][ T3356] __arm64_sys_write+0x68/0x98
[ 9426.926766][ T3356] do_el0_svc+0x124/0x220
[ 9426.930941][ T3356] el0_sync_handler+0x260/0x408
[ 9426.935634][ T3356] el0_sync+0x140/0x180
[ 9426.939633][ T3356]
[ 9426.941810][ T3356] Allocated by task 3356:
[ 9426.945985][ T3356] save_stack+0x24/0x50
[ 9426.949986][ T3356] __kasan_kmalloc.isra.13+0xc4/0xe0
[ 9426.955114][ T3356] kasan_kmalloc+0xc/0x18
[ 9426.959288][ T3356] kmem_cache_alloc_trace+0x1ec/0x318
[ 9426.964503][ T3356] arm_smmu_domain_alloc+0x54/0x148
[ 9426.969545][ T3356] iommu_group_alloc_default_domain+0xc0/0x440
[ 9426.975541][ T3356] iommu_probe_device+0x1c0/0x308
[ 9426.980409][ T3356] iort_iommu_configure+0x434/0x518
[ 9426.985452][ T3356] acpi_dma_configure+0xf0/0x128
[ 9426.990235][ T3356] pci_dma_configure+0x114/0x160
[ 9426.995017][ T3356] really_probe+0x124/0x6d8
[ 9426.999364][ T3356] driver_probe_device+0xc4/0x180
[ 9427.004232][ T3356] __device_attach_driver+0x184/0x1e8
[ 9427.009447][ T3356] bus_for_each_drv+0x114/0x1a0
[ 9427.014142][ T3356] __device_attach+0x19c/0x2a8
[ 9427.018749][ T3356] device_attach+0x10/0x18
[ 9427.023009][ T3356] pci_bus_add_device+0x70/0xf8
[ 9427.027704][ T3356] pci_iov_add_virtfn+0x7b4/0xb40
[ 9427.032571][ T3356] sriov_enable+0x5c8/0xc30
[ 9427.036918][ T3356] pci_enable_sriov+0x64/0x80
[ 9427.041485][ T3356] mlx5_core_sriov_configure+0x58/0x260 [mlx5_core]
[ 9427.047917][ T3356] sriov_numvfs_store+0x1c0/0x318
[ 9427.052784][ T3356] dev_attr_store+0x38/0x68
[ 9427.057131][ T3356] sysfs_kf_write+0xdc/0x128
[ 9427.061565][ T3356] kernfs_fop_write+0x23c/0x448
[ 9427.066260][ T3356] __vfs_write+0x54/0xe8
[ 9427.070346][ T3356] vfs_write+0x124/0x3f0
[ 9427.074433][ T3356] ksys_write+0xe8/0x1b8
[ 9427.078519][ T3356] __arm64_sys_write+0x68/0x98
[ 9427.083127][ T3356] do_el0_svc+0x124/0x220
[ 9427.087300][ T3356] el0_sync_handler+0x260/0x408
[ 9427.091994][ T3356] el0_sync+0x140/0x180
[ 9427.095992][ T3356]
[ 9427.098168][ T3356] Freed by task 3356:
[ 9427.101995][ T3356] save_stack+0x24/0x50
[ 9427.105996][ T3356] __kasan_slab_free+0x124/0x198
[ 9427.110777][ T3356] kasan_slab_free+0x10/0x18
[ 9427.115210][ T3356] slab_free_freelist_hook+0x110/0x298
[ 9427.120512][ T3356] kfree+0x128/0x668
[ 9427.124252][ T3356] arm_smmu_domain_free+0xf4/0x1a0
[ 9427.129206][ T3356] iommu_group_release+0xec/0x160
[ 9427.134074][ T3356] kobject_put+0xf4/0x238
[ 9427.138247][ T3356] kobject_del+0x110/0x190
[ 9427.142507][ T3356] kobject_put+0x1e4/0x238
[ 9427.146767][ T3356] iommu_group_remove_device+0x394/0x938
[ 9427.152242][ T3356] iommu_release_device+0x9c/0x178
iommu_release_device at drivers/iommu/iommu.c:300
[ 9427.157196][ T3356] iommu_bus_notifier+0x118/0x160
[ 9427.162065][ T3356] notifier_call_chain+0xa4/0x128
[ 9427.166934][ T3356] __blocking_notifier_call_chain+0x70/0xa8
[ 9427.172670][ T3356] blocking_notifier_call_chain+0x14/0x20
[ 9427.178233][ T3356] device_del+0x618/0xa00
[ 9427.182406][ T3356] pci_remove_bus_device+0x108/0x2d8
[ 9427.187535][ T3356] pci_stop_and_remove_bus_device+0x1c/0x28
[ 9427.193271][ T3356] pci_iov_remove_virtfn+0x228/0x368
[ 9427.198399][ T3356] sriov_disable+0x8c/0x348
[ 9427.202746][ T3356] pci_disable_sriov+0x5c/0x70
[ 9427.207398][ T3356] mlx5_core_sriov_configure+0xd8/0x260 [mlx5_core]
[ 9427.213830][ T3356] sriov_numvfs_store+0x240/0x318
[ 9427.218698][ T3356] dev_attr_store+0x38/0x68
[ 9427.223045][ T3356] sysfs_kf_write+0xdc/0x128
[ 9427.227478][ T3356] kernfs_fop_write+0x23c/0x448
[ 9427.232173][ T3356] __vfs_write+0x54/0xe8
[ 9427.236259][ T3356] vfs_write+0x124/0x3f0
[ 9427.240346][ T3356] ksys_write+0xe8/0x1b8
[ 9427.244433][ T3356] __arm64_sys_write+0x68/0x98
[ 9427.249041][ T3356] do_el0_svc+0x124/0x220
[ 9427.253215][ T3356] el0_sync_handler+0x260/0x408
[ 9427.257908][ T3356] el0_sync+0x140/0x180
[ 9427.261907][ T3356]
[ 9427.264084][ T3356] The buggy address belongs to the object at ffff0089df1a6e00
[ 9427.264084][ T3356] which belongs to the cache kmalloc-512 of size 512
[ 9427.277980][ T3356] The buggy address is located 360 bytes inside of
[ 9427.277980][ T3356] 512-byte region [ffff0089df1a6e00, ffff0089df1a7000)
[ 9427.291094][ T3356] The buggy address belongs to the page:
[ 9427.296571][ T3356] page:ffffffe02257c680 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff0089df1a1400
[ 9427.306823][ T3356] flags: 0x7ffff800000200(slab)
[ 9427.311520][ T3356] raw: 007ffff800000200 ffffffe02246b8c8 ffffffe02257ff88 ffff000000320680
[ 9427.319949][ T3356] raw: ffff0089df1a1400 00000000002a000e 00000001ffffffff ffff0089df1a5001
[ 9427.328374][ T3356] page dumped because: kasan: bad access detected
[ 9427.334630][ T3356] page->mem_cgroup:ffff0089df1a5001
[ 9427.339670][ T3356]
[ 9427.341846][ T3356] Memory state around the buggy address:
[ 9427.347322][ T3356] ffff0089df1a6e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.355228][ T3356] ffff0089df1a6e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.363133][ T3356] >ffff0089df1a6f00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.371038][ T3356] ^
[ 9427.378337][ T3356] ffff0089df1a6f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 9427.386242][ T3356] ffff0089df1a7000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 9427.394146][ T3356] ==================================================================
[ 9427.402052][ T3356] Disabling lock debugging due to kernel taint
>
> Thanks,
>
> Joerg
>
> Joerg Roedel (33):
> iommu: Move default domain allocation to separate function
> iommu/amd: Implement iommu_ops->def_domain_type call-back
> iommu/vt-d: Wire up iommu_ops->def_domain_type
> iommu/amd: Remove dma_mask check from check_device()
> iommu/amd: Return -ENODEV in add_device when device is not handled by
> IOMMU
> iommu: Add probe_device() and release_device() call-backs
> iommu: Move default domain allocation to iommu_probe_device()
> iommu: Keep a list of allocated groups in __iommu_probe_device()
> iommu: Move new probe_device path to separate function
> iommu: Split off default domain allocation from group assignment
> iommu: Move iommu_group_create_direct_mappings() out of
> iommu_group_add_device()
> iommu: Export bus_iommu_probe() and make is safe for re-probing
> iommu/amd: Remove dev_data->passthrough
> iommu/amd: Convert to probe/release_device() call-backs
> iommu/vt-d: Convert to probe/release_device() call-backs
> iommu/arm-smmu: Convert to probe/release_device() call-backs
> iommu/pamu: Convert to probe/release_device() call-backs
> iommu/s390: Convert to probe/release_device() call-backs
> iommu/virtio: Convert to probe/release_device() call-backs
> iommu/msm: Convert to probe/release_device() call-backs
> iommu/mediatek: Convert to probe/release_device() call-backs
> iommu/mediatek-v1 Convert to probe/release_device() call-backs
> iommu/qcom: Convert to probe/release_device() call-backs
> iommu/rockchip: Convert to probe/release_device() call-backs
> iommu/tegra: Convert to probe/release_device() call-backs
> iommu/renesas: Convert to probe/release_device() call-backs
> iommu/omap: Remove orphan_dev tracking
> iommu/omap: Convert to probe/release_device() call-backs
> iommu/exynos: Use first SYSMMU in controllers list for IOMMU core
> iommu/exynos: Convert to probe/release_device() call-backs
> iommu: Remove add_device()/remove_device() code-paths
> iommu: Move more initialization to __iommu_probe_device()
> iommu: Unexport iommu_group_get_for_dev()
>
> Sai Praneeth Prakhya (1):
> iommu: Add def_domain_type() callback in iommu_ops
>
> drivers/iommu/amd_iommu.c | 97 ++++----
> drivers/iommu/amd_iommu_types.h | 1 -
> drivers/iommu/arm-smmu-v3.c | 38 +---
> drivers/iommu/arm-smmu.c | 39 ++--
> drivers/iommu/exynos-iommu.c | 24 +-
> drivers/iommu/fsl_pamu_domain.c | 22 +-
> drivers/iommu/intel-iommu.c | 68 +-----
> drivers/iommu/iommu.c | 387 +++++++++++++++++++++++++-------
> drivers/iommu/ipmmu-vmsa.c | 60 ++---
> drivers/iommu/msm_iommu.c | 34 +--
> drivers/iommu/mtk_iommu.c | 24 +-
> drivers/iommu/mtk_iommu_v1.c | 50 ++---
> drivers/iommu/omap-iommu.c | 99 ++------
> drivers/iommu/qcom_iommu.c | 24 +-
> drivers/iommu/rockchip-iommu.c | 26 +--
> drivers/iommu/s390-iommu.c | 22 +-
> drivers/iommu/tegra-gart.c | 24 +-
> drivers/iommu/tegra-smmu.c | 31 +--
> drivers/iommu/virtio-iommu.c | 41 +---
> include/linux/iommu.h | 21 +-
> 20 files changed, 531 insertions(+), 601 deletions(-)
>
> --
> 2.17.1
>
> _______________________________________________
> iommu mailing list
> iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx
> https://lists.linuxfoundation.org/mailman/listinfo/iommu