Re: "Convert the AMD iommu driver to the dma-iommu api" is buggy

[Qian Cai]

On Wed, 2019-10-16 at 10:55 -0400, Qian Cai wrote:
> Today's linux-next generates a lot of warnings on multiple servers during boot
> due to the series "iommu/amd: Convert the AMD iommu driver to the dma-iommu api"
> [1]. Reverted the whole things fixed them.
> 
> [1] https://lore.kernel.org/lkml/20190908165642.22253-1-murphyt7@xxxxxx/
> 

BTW, the previous x86 warning was from only reverted one patch "iommu: Add gfp
parameter to iommu_ops::map" where proved to be insufficient. Now, pasting the
correct warning.

[ÂÂ564.365768][ T6222] BUG: sleeping function called from invalid context at
mm/page_alloc.c:4692
[ÂÂ564.374447][ T6222] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid:
6222, name: git
[ÂÂ564.382969][ T6222] INFO: lockdep is turned off.
[ÂÂ564.387644][ T6222] CPU: 25 PID: 6222 Comm: git Tainted:
GÂÂÂÂÂÂÂÂWÂÂÂÂÂÂÂÂÂ5.4.0-rc3-next-20191016 #6
[ÂÂ564.397011][ T6222] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385
Gen10, BIOS A40 07/10/2019
[ÂÂ564.406291][ T6222] Call Trace:
[ÂÂ564.409470][ T6222]ÂÂdump_stack+0x86/0xca
[ÂÂ564.413517][ T6222]ÂÂ___might_sleep.cold.92+0xd2/0x122
[ÂÂ564.418694][ T6222]ÂÂ__might_sleep+0x73/0xe0
[ÂÂ564.422999][ T6222]ÂÂ__alloc_pages_nodemask+0x442/0x720
[ÂÂ564.428265][ T6222]ÂÂ? __alloc_pages_slowpath+0x18d0/0x18d0
[ÂÂ564.433883][ T6222]ÂÂ? arch_stack_walk+0x7f/0xf0
[ÂÂ564.438534][ T6222]ÂÂ? create_object+0x4a2/0x540
[ÂÂ564.443188][ T6222]ÂÂalloc_pages_current+0x9c/0x110
[ÂÂ564.448098][ T6222]ÂÂ__get_free_pages+0x12/0x60
[ÂÂ564.452659][ T6222]ÂÂget_zeroed_page+0x16/0x20
[ÂÂ564.457137][ T6222]ÂÂamd_iommu_map+0x504/0x850
[ÂÂ564.461612][ T6222]ÂÂ? amd_iommu_domain_direct_map+0x60/0x60
[ÂÂ564.467312][ T6222]ÂÂ? lockdep_hardirqs_on+0x16/0x2a0
[ÂÂ564.472400][ T6222]ÂÂ? alloc_iova+0x189/0x210
[ÂÂ564.476790][ T6222]ÂÂ__iommu_map+0x1c1/0x4e0
[ÂÂ564.481090][ T6222]ÂÂ? iommu_get_dma_domain+0x40/0x40
[ÂÂ564.486181][ T6222]ÂÂ? alloc_iova_fast+0x258/0x3d1
[ÂÂ564.491009][ T6222]ÂÂ? create_object+0x4a2/0x540
[ÂÂ564.495656][ T6222]ÂÂ__iommu_map_sg+0xa5/0x130
[ÂÂ564.500135][ T6222]ÂÂiommu_map_sg_atomic+0x14/0x20
[ÂÂ564.504958][ T6222]ÂÂiommu_dma_map_sg+0x2c3/0x450
[ÂÂ564.509699][ T6222]ÂÂscsi_dma_map+0xd7/0x160
[ÂÂ564.514010][ T6222]ÂÂpqi_raid_submit_scsi_cmd_with_io_request+0x392/0x420
[smartpqi]
[ÂÂ564.521811][ T6222]ÂÂ? pqi_alloc_io_request+0x127/0x140 [smartpqi]
[ÂÂ564.528037][ T6222]ÂÂpqi_scsi_queue_command+0x8ab/0xe00 [smartpqi]
[ÂÂ564.534264][ T6222]ÂÂ? pqi_eh_device_reset_handler+0x9c0/0x9c0 [smartpqi]
[ÂÂ564.541100][ T6222]ÂÂ? sd_init_command+0xa25/0x1346 [sd_mod]
[ÂÂ564.546802][ T6222]ÂÂscsi_queue_rq+0xd19/0x1360
[ÂÂ564.551372][ T6222]ÂÂ__blk_mq_try_issue_directly+0x295/0x3f0
[ÂÂ564.557071][ T6222]ÂÂ? blk_mq_request_bypass_insert+0xd0/0xd0
[ÂÂ564.562860][ T6222]ÂÂ? debug_lockdep_rcu_enabled+0x27/0x60
[ÂÂ564.568384][ T6222]ÂÂblk_mq_try_issue_directly+0xad/0x130
[ÂÂ564.573821][ T6222]ÂÂ? __blk_mq_try_issue_directly+0x3f0/0x3f0
[ÂÂ564.579693][ T6222]ÂÂ? blk_add_rq_to_plug+0xcd/0x110
[ÂÂ564.584693][ T6222]ÂÂblk_mq_make_request+0xcee/0x1120
[ÂÂ564.589777][ T6222]ÂÂ? lock_downgrade+0x3c0/0x3c0
[ÂÂ564.594517][ T6222]ÂÂ? blk_mq_try_issue_directly+0x130/0x130
[ÂÂ564.600218][ T6222]ÂÂ? blk_queue_enter+0x78d/0x810
[ÂÂ564.605041][ T6222]ÂÂ? generic_make_request_checks+0xf30/0xf30
[ÂÂ564.610915][ T6222]ÂÂ? lock_downgrade+0x3c0/0x3c0
[ÂÂ564.615655][ T6222]ÂÂ? __srcu_read_unlock+0x24/0x50
[ÂÂ564.620565][ T6222]ÂÂ? generic_make_request+0x150/0x650
[ÂÂ564.625833][ T6222]ÂÂgeneric_make_request+0x196/0x650
[ÂÂ564.630921][ T6222]ÂÂ? blk_queue_enter+0x810/0x810
[ÂÂ564.635747][ T6222]ÂÂsubmit_bio+0xaa/0x270
[ÂÂ564.639873][ T6222]ÂÂ? submit_bio+0xaa/0x270
[ÂÂ564.644172][ T6222]ÂÂ? generic_make_request+0x650/0x650
[ÂÂ564.649437][ T6222]ÂÂ? iomap_readpage+0x260/0x260
[ÂÂ564.654173][ T6222]ÂÂiomap_readpages+0x154/0x3d0
[ÂÂ564.658823][ T6222]ÂÂ? iomap_zero_range_actor+0x330/0x330
[ÂÂ564.664257][ T6222]ÂÂ? __might_sleep+0x73/0xe0
[ÂÂ564.668836][ T6222]ÂÂxfs_vm_readpages+0xaf/0x1f0 [xfs]
[ÂÂ564.674016][ T6222]ÂÂread_pages+0xe2/0x3b0
[ÂÂ564.678142][ T6222]ÂÂ? read_cache_pages+0x350/0x350
[ÂÂ564.683057][ T6222]ÂÂ? __page_cache_alloc+0x12c/0x230
[ÂÂ564.688148][ T6222]ÂÂ__do_page_cache_readahead+0x346/0x3a0
[ÂÂ564.693670][ T6222]ÂÂ? read_pages+0x3b0/0x3b0
[ÂÂ564.698059][ T6222]ÂÂ? lockdep_hardirqs_on+0x16/0x2a0
[ÂÂ564.703247][ T6222]ÂÂ? __xfs_filemap_fault+0x167/0x4a0 [xfs]
[ÂÂ564.708947][ T6222]ÂÂfilemap_fault+0xa13/0xe70
[ÂÂ564.713528][ T6222]ÂÂ__xfs_filemap_fault+0x167/0x4a0 [xfs]
[ÂÂ564.719059][ T6222]ÂÂ? kmemleak_alloc+0x57/0x90
[ÂÂ564.723724][ T6222]ÂÂ? xfs_file_read_iter+0x3c0/0x3c0 [xfs]
[ÂÂ564.729337][ T6222]ÂÂ? debug_check_no_locks_freed+0x2c/0xe0
[ÂÂ564.734946][ T6222]ÂÂ? lockdep_init_map+0x8b/0x2b0
[ÂÂ564.739872][ T6222]ÂÂxfs_filemap_fault+0x68/0x70 [xfs]
[ÂÂ564.745046][ T6222]ÂÂ__do_fault+0x83/0x220
[ÂÂ564.749172][ T6222]ÂÂ__handle_mm_fault+0xd76/0x1f40
[ÂÂ564.754084][ T6222]ÂÂ? __pmd_alloc+0x280/0x280
[ÂÂ564.758559][ T6222]ÂÂ? debug_lockdep_rcu_enabled+0x27/0x60

> 
> [ÂÂ183.553150] BUG: sleeping function called from invalid context at
> drivers/iommu/iommu.c:1950
> [ÂÂ183.562306] in_atomic(): 1, irqs_disabled(): 128, non_block: 0, pid: 1486,
> name: kworker/0:3
> [ÂÂ183.571450] 5 locks held by kworker/0:3/1486:
> [ÂÂ183.576510]ÂÂ#0: 44ff0008000ce128 ((wq_completion)events){+.+.}, at:
> process_one_work+0x25c/0x948
> [ÂÂ183.586110]ÂÂ#1: 43ff00081fb2fcf8 ((work_completion)(&wfc.work)){+.+.}, at:
> process_one_work+0x280/0x948
> [ÂÂ183.596310]ÂÂ#2: ffff000a2c661a08 (&dev->intf_state_mutex){+.+.}, at:
> mlx5_load_one+0x68/0x12e0 [mlx5_core]
> [ÂÂ183.606916]ÂÂ#3: ffff9000127e4560 (irq_domain_mutex){+.+.}, at:
> __irq_domain_alloc_irqs+0x1f8/0x430
> [ÂÂ183.616683]ÂÂ#4: 02ff0095ca0ed8f0 (&(&cookie->msi_lock)->rlock){....}, at:
> iommu_dma_prepare_msi+0x70/0x210
> [ÂÂ183.627146] irq event stamp: 378872
> [ÂÂ183.631345] hardirqs lastÂÂenabled at (378871): [<ffff9000109d0230>]
> _raw_write_unlock_irqrestore+0x4c/0x84
> [ÂÂ183.641791] hardirqs last disabled at (378872): [<ffff9000109cf7a0>]
> _raw_spin_lock_irqsave+0x38/0x9c
> [ÂÂ183.651717] softirqs lastÂÂenabled at (377854): [<ffff9000100824f4>]
> __do_softirq+0x864/0x900
> [ÂÂ183.660951] softirqs last disabled at (377841): [<ffff900010118768>]
> irq_exit+0x1c8/0x238
> [ÂÂ183.669836] CPU: 0 PID: 1486 Comm: kworker/0:3 Tainted:
> GÂÂÂÂÂÂÂÂWÂÂÂÂLÂÂÂÂ5.4.0-rc3-next-20191016+ #8
> [ÂÂ183.679845] Hardware name: HPE Apollo 70ÂÂÂÂÂÂÂÂÂÂÂÂÂ/C01_APACHE_MBÂÂÂÂÂÂÂÂÂ,
> BIOS L50_5.13_1.11 06/18/2019
> [ÂÂ183.690292] Workqueue: events work_for_cpu_fn
> [ÂÂ183.695357] Call trace:
> [ÂÂ183.698510]ÂÂdump_backtrace+0x0/0x248
> [ÂÂ183.702878]ÂÂshow_stack+0x20/0x2c
> [ÂÂ183.706900]ÂÂdump_stack+0xc8/0x130
> [ÂÂ183.711009]ÂÂ___might_sleep+0x314/0x328
> [ÂÂ183.715551]ÂÂ__might_sleep+0x7c/0xe0
> [ÂÂ183.719832]ÂÂiommu_map+0x40/0x70
> [ÂÂ183.723766]ÂÂiommu_dma_prepare_msi+0x16c/0x210
> [ÂÂ183.728916]ÂÂits_irq_domain_alloc+0x100/0x254
> [ÂÂ183.733979]ÂÂirq_domain_alloc_irqs_parent+0x74/0x90
> [ÂÂ183.739562]ÂÂmsi_domain_alloc+0xa0/0x170
> [ÂÂ183.744190]ÂÂ__irq_domain_alloc_irqs+0x228/0x430
> [ÂÂ183.749512]ÂÂmsi_domain_alloc_irqs+0x130/0x548
> [ÂÂ183.754663]ÂÂpci_msi_setup_msi_irqs+0x64/0x74
> [ÂÂ183.759726]ÂÂ__pci_enable_msix_range+0x52c/0x878
> [ÂÂ183.765049]ÂÂpci_alloc_irq_vectors_affinity+0x94/0x168
> [ÂÂ183.771028]ÂÂmlx5_irq_table_create+0x178/0x748 [mlx5_core]
> [ÂÂ183.777353]ÂÂmlx5_load_one+0x710/0x12e0 [mlx5_core]
> [ÂÂ183.783069]ÂÂinit_one+0x514/0x898 [mlx5_core]
> [ÂÂ183.788134]ÂÂlocal_pci_probe+0x74/0xcc
> [ÂÂ183.792589]ÂÂwork_for_cpu_fn+0x30/0x4c
> [ÂÂ183.797045]ÂÂprocess_one_work+0x4f4/0x948
> [ÂÂ183.801760]ÂÂprocess_scheduled_works+0x34/0x54
> [ÂÂ183.806909]ÂÂworker_thread+0x348/0x4bc
> [ÂÂ183.811364]ÂÂkthread+0x1cc/0x1e8
> [ÂÂ183.815299]ÂÂret_from_fork+0x10/0x18
> [ÂÂ184.621631] mlx5_core 0000:0b:00.1: Port module event: module 1, Cable
> unplugged
> [ÂÂ184.867367] mlx5_core 0000:0b:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256)
> RxCqeCmprss(0)
> [ÂÂ186.181802] mlx5_core 0000:0b:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256)
> RxCqeCmprss(0)