Re: [PATCH v6 00/22] Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers

From: Robin Murphy
Date: Tue Jun 28 2022 - 08:31:18 EST

On 2022-05-27 00:50, Dmitry Osipenko wrote:
Hello,

This patchset introduces memory shrinker for the VirtIO-GPU DRM driver
and adds memory purging and eviction support to VirtIO-GPU driver.

The new dma-buf locking convention is introduced here as well.

During OOM, the shrinker will release BOs that are marked as "not needed"
by userspace using the new madvise IOCTL, it will also evict idling BOs
to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark
the cached BOs as "not needed", allowing kernel driver to release memory
of the cached shmem BOs on lowmem situations, preventing OOM kills.

The Panfrost driver is switched to use generic memory shrinker.

I think we still have some outstanding issues here - Alyssa reported some weirdness yesterday, so I just tried provoking a low-memory condition locally with this series applied and a few debug options enabled, and the results as below were... interesting.

Thanks,
Robin.

----->8-----
[ 68.295951] ======================================================
[ 68.295956] WARNING: possible circular locking dependency detected
[ 68.295963] 5.19.0-rc3+ #400 Not tainted
[ 68.295972] ------------------------------------------------------
[ 68.295977] cc1/295 is trying to acquire lock:
[ 68.295986] ffff000008d7f1a0 (reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_gem_shmem_free+0x7c/0x198
[ 68.296036]
[ 68.296036] but task is already holding lock:
[ 68.296041] ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at: __alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296080]
[ 68.296080] which lock already depends on the new lock.
[ 68.296080]
[ 68.296085]
[ 68.296085] the existing dependency chain (in reverse order) is:
[ 68.296090]
[ 68.296090] -> #1 (fs_reclaim){+.+.}-{0:0}:
[ 68.296111] fs_reclaim_acquire+0xb8/0x150
[ 68.296130] dma_resv_lockdep+0x298/0x3fc
[ 68.296148] do_one_initcall+0xe4/0x5f8
[ 68.296163] kernel_init_freeable+0x414/0x49c
[ 68.296180] kernel_init+0x2c/0x148
[ 68.296195] ret_from_fork+0x10/0x20
[ 68.296207]
[ 68.296207] -> #0 (reservation_ww_class_mutex){+.+.}-{3:3}:
[ 68.296229] __lock_acquire+0x1724/0x2398
[ 68.296246] lock_acquire+0x218/0x5b0
[ 68.296260] __ww_mutex_lock.constprop.0+0x158/0x2378
[ 68.296277] ww_mutex_lock+0x7c/0x4d8
[ 68.296291] drm_gem_shmem_free+0x7c/0x198
[ 68.296304] panfrost_gem_free_object+0x118/0x138
[ 68.296318] drm_gem_object_free+0x40/0x68
[ 68.296334] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8
[ 68.296352] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170
[ 68.296368] do_shrink_slab+0x220/0x808
[ 68.296381] shrink_slab+0x11c/0x408
[ 68.296392] shrink_node+0x6ac/0xb90
[ 68.296403] do_try_to_free_pages+0x1dc/0x8d0
[ 68.296416] try_to_free_pages+0x1ec/0x5b0
[ 68.296429] __alloc_pages_slowpath.constprop.0+0x528/0x1470
[ 68.296444] __alloc_pages+0x4e0/0x5b8
[ 68.296455] __folio_alloc+0x24/0x60
[ 68.296467] vma_alloc_folio+0xb8/0x2f8
[ 68.296483] alloc_zeroed_user_highpage_movable+0x58/0x68
[ 68.296498] __handle_mm_fault+0x918/0x12a8
[ 68.296513] handle_mm_fault+0x130/0x300
[ 68.296527] do_page_fault+0x1d0/0x568
[ 68.296539] do_translation_fault+0xa0/0xb8
[ 68.296551] do_mem_abort+0x68/0xf8
[ 68.296562] el0_da+0x74/0x100
[ 68.296572] el0t_64_sync_handler+0x68/0xc0
[ 68.296585] el0t_64_sync+0x18c/0x190
[ 68.296596]
[ 68.296596] other info that might help us debug this:
[ 68.296596]
[ 68.296601] Possible unsafe locking scenario:
[ 68.296601]
[ 68.296604] CPU0 CPU1
[ 68.296608] ---- ----
[ 68.296612] lock(fs_reclaim);
[ 68.296622] lock(reservation_ww_class_mutex);
[ 68.296633] lock(fs_reclaim);
[ 68.296644] lock(reservation_ww_class_mutex);
[ 68.296654]
[ 68.296654] *** DEADLOCK ***
[ 68.296654]
[ 68.296658] 3 locks held by cc1/295:
[ 68.296666] #0: ffff00000616e898 (&mm->mmap_lock){++++}-{3:3}, at: do_page_fault+0x144/0x568
[ 68.296702] #1: ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at: __alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296740] #2: ffff80000c1215b0 (shrinker_rwsem){++++}-{3:3}, at: shrink_slab+0xc0/0x408
[ 68.296774]
[ 68.296774] stack backtrace:
[ 68.296780] CPU: 2 PID: 295 Comm: cc1 Not tainted 5.19.0-rc3+ #400
[ 68.296794] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 68.296803] Call trace:
[ 68.296808] dump_backtrace+0x1e4/0x1f0
[ 68.296821] show_stack+0x20/0x70
[ 68.296832] dump_stack_lvl+0x8c/0xb8
[ 68.296849] dump_stack+0x1c/0x38
[ 68.296864] print_circular_bug.isra.0+0x284/0x378
[ 68.296881] check_noncircular+0x1d8/0x1f8
[ 68.296896] __lock_acquire+0x1724/0x2398
[ 68.296911] lock_acquire+0x218/0x5b0
[ 68.296926] __ww_mutex_lock.constprop.0+0x158/0x2378
[ 68.296942] ww_mutex_lock+0x7c/0x4d8
[ 68.296956] drm_gem_shmem_free+0x7c/0x198
[ 68.296970] panfrost_gem_free_object+0x118/0x138
[ 68.296984] drm_gem_object_free+0x40/0x68
[ 68.296999] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8
[ 68.297017] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170
[ 68.297033] do_shrink_slab+0x220/0x808
[ 68.297045] shrink_slab+0x11c/0x408
[ 68.297056] shrink_node+0x6ac/0xb90
[ 68.297068] do_try_to_free_pages+0x1dc/0x8d0
[ 68.297081] try_to_free_pages+0x1ec/0x5b0
[ 68.297094] __alloc_pages_slowpath.constprop.0+0x528/0x1470
[ 68.297110] __alloc_pages+0x4e0/0x5b8
[ 68.297122] __folio_alloc+0x24/0x60
[ 68.297134] vma_alloc_folio+0xb8/0x2f8
[ 68.297148] alloc_zeroed_user_highpage_movable+0x58/0x68
[ 68.297163] __handle_mm_fault+0x918/0x12a8
[ 68.297178] handle_mm_fault+0x130/0x300
[ 68.297193] do_page_fault+0x1d0/0x568
[ 68.297205] do_translation_fault+0xa0/0xb8
[ 68.297218] do_mem_abort+0x68/0xf8
[ 68.297229] el0_da+0x74/0x100
[ 68.297239] el0t_64_sync_handler+0x68/0xc0
[ 68.297252] el0t_64_sync+0x18c/0x190
[ 68.471812] arm-scmi firmware:scmi: timed out in resp(caller: scmi_power_state_set+0x11c/0x190)
[ 68.501947] arm-scmi firmware:scmi: Message for 119 type 0 is not expected!
[ 68.939686] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000915e2d34
[ 69.739386] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 70.415329] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ee980c7e
[ 70.987166] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ffb7ff37
[ 71.914939] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000e92b26e
[ 72.426987] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 73.578683] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 74.090555] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000075d00f9
[ 74.922709] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000005add546
[ 75.434401] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000154189b
[ 76.394300] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 76.906236] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ee980c7e
[ 79.657234] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000f6d059fb
[ 80.168831] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000061a0f6bf
[ 80.808354] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000071ade02
[ 81.319967] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b0afea73
[ 81.831574] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000d78f36c2
[ 82.343160] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000f689397
[ 83.046689] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000412c2a2f
[ 83.558352] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000020e551b3
[ 84.261913] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000009437aace
[ 84.773576] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 85.317275] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 85.829035] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000e92b26e
[ 86.660555] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 87.172126] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b940e406
[ 87.875846] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 88.387443] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000009437aace
[ 89.059175] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000075dadb7f
[ 89.570960] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000005add546
[ 90.146687] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000cba2873c
[ 90.662497] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a4beb490
[ 95.392748] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000005b5fc4ec
[ 95.904179] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a17436ee
[ 96.416085] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000003888d2a7
[ 96.927874] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000093e04a98
[ 97.439742] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 97.954109] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000084c51113
[ 98.467374] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000664663ce
[ 98.975192] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000060f2d45c
[ 99.487231] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b29288f8
[ 99.998833] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000002f07ab24
[ 100.510744] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000008c15c751
[ 100.511411] ==================================================================
[ 100.511419] BUG: KASAN: use-after-free in irq_work_single+0xa4/0x110
[ 100.511445] Write of size 4 at addr ffff0000107f5830 by task glmark2-es2-drm/280
[ 100.511458]
[ 100.511464] CPU: 1 PID: 280 Comm: glmark2-es2-drm Not tainted 5.19.0-rc3+ #400
[ 100.511479] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 100.511489] Call trace:
[ 100.511494] dump_backtrace+0x1e4/0x1f0
[ 100.511512] show_stack+0x20/0x70
[ 100.511523] dump_stack_lvl+0x8c/0xb8
[ 100.511543] print_report+0x16c/0x668
[ 100.511559] kasan_report+0x80/0x208
[ 100.511574] kasan_check_range+0x100/0x1b8
[ 100.511590] __kasan_check_write+0x34/0x60
[ 100.511607] irq_work_single+0xa4/0x110
[ 100.511619] irq_work_run_list+0x6c/0x88
[ 100.511632] irq_work_run+0x28/0x48
[ 100.511644] ipi_handler+0x254/0x468
[ 100.511664] handle_percpu_devid_irq+0x11c/0x518
[ 100.511681] generic_handle_domain_irq+0x50/0x70
[ 100.511699] gic_handle_irq+0xd4/0x118
[ 100.511711] call_on_irq_stack+0x2c/0x58
[ 100.511725] do_interrupt_handler+0xc0/0xc8
[ 100.511741] el1_interrupt+0x40/0x68
[ 100.511754] el1h_64_irq_handler+0x18/0x28
[ 100.511767] el1h_64_irq+0x64/0x68
[ 100.511778] irq_work_queue+0xc0/0xd8
[ 100.511790] drm_sched_entity_fini+0x2c4/0x3b0
[ 100.511805] drm_sched_entity_destroy+0x2c/0x40
[ 100.511818] panfrost_job_close+0x44/0x1c0
[ 100.511833] panfrost_postclose+0x38/0x60
[ 100.511845] drm_file_free.part.0+0x33c/0x4b8
[ 100.511862] drm_close_helper.isra.0+0xc0/0xd8
[ 100.511877] drm_release+0xe4/0x1e0
[ 100.511891] __fput+0xf8/0x390
[ 100.511904] ____fput+0x18/0x28
[ 100.511917] task_work_run+0xc4/0x1e0
[ 100.511929] do_exit+0x554/0x1168
[ 100.511945] do_group_exit+0x60/0x108
[ 100.511960] __arm64_sys_exit_group+0x34/0x38
[ 100.511977] invoke_syscall+0x64/0x180
[ 100.511993] el0_svc_common.constprop.0+0x13c/0x170
[ 100.512012] do_el0_svc+0x48/0xe8
[ 100.512028] el0_svc+0x5c/0xe0
[ 100.512038] el0t_64_sync_handler+0xb8/0xc0
[ 100.512051] el0t_64_sync+0x18c/0x190
[ 100.512064]
[ 100.512068] Allocated by task 280:
[ 100.512075] kasan_save_stack+0x2c/0x58
[ 100.512091] __kasan_kmalloc+0x90/0xb8
[ 100.512105] kmem_cache_alloc_trace+0x1d4/0x330
[ 100.512118] panfrost_ioctl_submit+0x100/0x630
[ 100.512131] drm_ioctl_kernel+0x160/0x250
[ 100.512147] drm_ioctl+0x36c/0x628
[ 100.512161] __arm64_sys_ioctl+0xd8/0x120
[ 100.512178] invoke_syscall+0x64/0x180
[ 100.512194] el0_svc_common.constprop.0+0x13c/0x170
[ 100.512211] do_el0_svc+0x48/0xe8
[ 100.512226] el0_svc+0x5c/0xe0
[ 100.512236] el0t_64_sync_handler+0xb8/0xc0
[ 100.512248] el0t_64_sync+0x18c/0x190
[ 100.512259]
[ 100.512262] Freed by task 280:
[ 100.512268] kasan_save_stack+0x2c/0x58
[ 100.512283] kasan_set_track+0x2c/0x40
[ 100.512296] kasan_set_free_info+0x28/0x50
[ 100.512312] __kasan_slab_free+0xf0/0x170
[ 100.512326] kfree+0x124/0x418
[ 100.512337] panfrost_job_cleanup+0x1f0/0x298
[ 100.512350] panfrost_job_free+0x80/0xb0
[ 100.512363] drm_sched_entity_kill_jobs_irq_work+0x80/0xa0
[ 100.512377] irq_work_single+0x88/0x110
[ 100.512389] irq_work_run_list+0x6c/0x88
[ 100.512401] irq_work_run+0x28/0x48
[ 100.512413] ipi_handler+0x254/0x468
[ 100.512427] handle_percpu_devid_irq+0x11c/0x518
[ 100.512443] generic_handle_domain_irq+0x50/0x70
[ 100.512460] gic_handle_irq+0xd4/0x118
[ 100.512471]
[ 100.512474] The buggy address belongs to the object at ffff0000107f5800
[ 100.512474] which belongs to the cache kmalloc-512 of size 512
[ 100.512484] The buggy address is located 48 bytes inside of
[ 100.512484] 512-byte region [ffff0000107f5800, ffff0000107f5a00)
[ 100.512497]
[ 100.512500] The buggy address belongs to the physical page:
[ 100.512506] page:000000000a626feb refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x907f4
[ 100.512520] head:000000000a626feb order:2 compound_mapcount:0 compound_pincount:0
[ 100.512530] flags: 0xffff00000010200(slab|head|node=0|zone=0|lastcpupid=0xffff)
[ 100.512556] raw: 0ffff00000010200 fffffc0000076400 dead000000000002 ffff000000002600
[ 100.512569] raw: 0000000000000000 0000000080100010 00000001ffffffff 0000000000000000
[ 100.512577] page dumped because: kasan: bad access detected
[ 100.512582]
[ 100.512585] Memory state around the buggy address:
[ 100.512592] ffff0000107f5700: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 100.512602] ffff0000107f5780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 100.512612] >ffff0000107f5800: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 100.512619] ^
[ 100.512627] ffff0000107f5880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 100.512636] ffff0000107f5900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 100.512643] ==================================================================
[ 101.022573] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000be4b1b31
[ 101.534469] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a8ff2c8a
[ 101.535981] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:870
[ 101.535994] in_atomic(): 1, irqs_disabled(): 128, non_block: 0, pid: 280, name: glmark2-es2-drm
[ 101.536006] preempt_count: 10000, expected: 0
[ 101.536012] RCU nest depth: 0, expected: 0
[ 101.536019] INFO: lockdep is turned off.
[ 101.536023] irq event stamp: 1666508
[ 101.536029] hardirqs last enabled at (1666507): [<ffff80000997ed70>] exit_to_kernel_mode.isra.0+0x40/0x140
[ 101.536056] hardirqs last disabled at (1666508): [<ffff800009985030>] __schedule+0xb38/0xea8
[ 101.536076] softirqs last enabled at (1664950): [<ffff800008010ac8>] __do_softirq+0x6b8/0x89c
[ 101.536092] softirqs last disabled at (1664941): [<ffff8000080e4fdc>] irq_exit_rcu+0x27c/0x2b0
[ 101.536118] CPU: 1 PID: 280 Comm: glmark2-es2-drm Tainted: G B 5.19.0-rc3+ #400
[ 101.536134] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 101.536143] Call trace:
[ 101.536147] dump_backtrace+0x1e4/0x1f0
[ 101.536161] show_stack+0x20/0x70
[ 101.536171] dump_stack_lvl+0x8c/0xb8
[ 101.536189] dump_stack+0x1c/0x38
[ 101.536204] __might_resched+0x1f0/0x2b0
[ 101.536220] __might_sleep+0x74/0xd0
[ 101.536234] ww_mutex_lock+0x40/0x4d8
[ 101.536249] drm_gem_shmem_free+0x7c/0x198
[ 101.536264] panfrost_gem_free_object+0x118/0x138
[ 101.536278] drm_gem_object_free+0x40/0x68
[ 101.536295] panfrost_job_cleanup+0x1bc/0x298
[ 101.536309] panfrost_job_free+0x80/0xb0
[ 101.536322] drm_sched_entity_kill_jobs_irq_work+0x80/0xa0
[ 101.536337] irq_work_single+0x88/0x110
[ 101.536351] irq_work_run_list+0x6c/0x88
[ 101.536364] irq_work_run+0x28/0x48
[ 101.536375] ipi_handler+0x254/0x468
[ 101.536392] handle_percpu_devid_irq+0x11c/0x518
[ 101.536409] generic_handle_domain_irq+0x50/0x70
[ 101.536428] gic_handle_irq+0xd4/0x118
[ 101.536439] call_on_irq_stack+0x2c/0x58
[ 101.536453] do_interrupt_handler+0xc0/0xc8
[ 101.536468] el1_interrupt+0x40/0x68
[ 101.536479] el1h_64_irq_handler+0x18/0x28
[ 101.536492] el1h_64_irq+0x64/0x68
[ 101.536503] __asan_load8+0x30/0xd0
[ 101.536519] drm_sched_entity_fini+0x1e8/0x3b0
[ 101.536532] drm_sched_entity_destroy+0x2c/0x40
[ 101.536545] panfrost_job_close+0x44/0x1c0
[ 101.536559] panfrost_postclose+0x38/0x60
[ 101.536571] drm_file_free.part.0+0x33c/0x4b8
[ 101.536586] drm_close_helper.isra.0+0xc0/0xd8
[ 101.536601] drm_release+0xe4/0x1e0
[ 101.536615] __fput+0xf8/0x390
[ 101.536628] ____fput+0x18/0x28
[ 101.536640] task_work_run+0xc4/0x1e0
[ 101.536652] do_exit+0x554/0x1168
[ 101.536667] do_group_exit+0x60/0x108
[ 101.536682] __arm64_sys_exit_group+0x34/0x38
[ 101.536698] invoke_syscall+0x64/0x180
[ 101.536714] el0_svc_common.constprop.0+0x13c/0x170
[ 101.536733] do_el0_svc+0x48/0xe8
[ 101.536748] el0_svc+0x5c/0xe0
[ 101.536759] el0t_64_sync_handler+0xb8/0xc0
[ 101.536771] el0t_64_sync+0x18c/0x190
[ 101.541928] ------------[ cut here ]------------
[ 101.541934] kernel BUG at kernel/irq_work.c:235!
[ 101.541944] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 101.541961] Modules linked in:
[ 101.541978] CPU: 1 PID: 280 Comm: glmark2-es2-drm Tainted: G B W 5.19.0-rc3+ #400
[ 101.541997] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 101.542009] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 101.542027] pc : irq_work_run_list+0x80/0x88
[ 101.542044] lr : irq_work_run+0x34/0x48
[ 101.542060] sp : ffff80000da37eb0
[ 101.542069] x29: ffff80000da37eb0 x28: ffff000006bb0000 x27: ffff000006bb0008
[ 101.542107] x26: ffff80000da37f20 x25: ffff8000080304d8 x24: 0000000000000001
[ 101.542142] x23: ffff80000abcd008 x22: ffff80000da37ed0 x21: ffff80001c0de000
[ 101.542177] x20: ffff80000abcd008 x19: ffff80000abdbad0 x18: 0000000000000000
[ 101.542212] x17: 616e202c30383220 x16: 3a646970202c3020 x15: ffff8000082df9d0
[ 101.542246] x14: ffff800008dfada8 x13: 0000000000000003 x12: 1fffe000018b2a06
[ 101.542280] x11: ffff6000018b2a06 x10: dfff800000000000 x9 : ffff00000c595033
[ 101.542315] x8 : ffff6000018b2a07 x7 : 0000000000000001 x6 : 00000000000000fb
[ 101.542349] x5 : ffff00000c595030 x4 : 0000000000000000 x3 : ffff00000c595030
[ 101.542382] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000026cb9ad0
[ 101.542416] Call trace:
[ 101.542424] irq_work_run_list+0x80/0x88
[ 101.542441] ipi_handler+0x254/0x468
[ 101.542460] handle_percpu_devid_irq+0x11c/0x518
[ 101.542480] generic_handle_domain_irq+0x50/0x70
[ 101.542501] gic_handle_irq+0xd4/0x118
[ 101.542516] call_on_irq_stack+0x2c/0x58
[ 101.542534] do_interrupt_handler+0xc0/0xc8
[ 101.542553] el1_interrupt+0x40/0x68
[ 101.542568] el1h_64_irq_handler+0x18/0x28
[ 101.542584] el1h_64_irq+0x64/0x68
[ 101.542599] __asan_load8+0x30/0xd0
[ 101.542617] drm_sched_entity_fini+0x1e8/0x3b0
[ 101.542634] drm_sched_entity_destroy+0x2c/0x40
[ 101.542651] panfrost_job_close+0x44/0x1c0
[ 101.542669] panfrost_postclose+0x38/0x60
[ 101.542685] drm_file_free.part.0+0x33c/0x4b8
[ 101.542704] drm_close_helper.isra.0+0xc0/0xd8
[ 101.542723] drm_release+0xe4/0x1e0
[ 101.542740] __fput+0xf8/0x390
[ 101.542756] ____fput+0x18/0x28
[ 101.542773] task_work_run+0xc4/0x1e0
[ 101.542788] do_exit+0x554/0x1168
[ 101.542806] do_group_exit+0x60/0x108
[ 101.542825] __arm64_sys_exit_group+0x34/0x38
[ 101.542845] invoke_syscall+0x64/0x180
[ 101.542865] el0_svc_common.constprop.0+0x13c/0x170
[ 101.542887] do_el0_svc+0x48/0xe8
[ 101.542906] el0_svc+0x5c/0xe0
[ 101.542921] el0t_64_sync_handler+0xb8/0xc0
[ 101.542938] el0t_64_sync+0x18c/0x190
[ 101.542960] Code: a94153f3 a8c27bfd d50323bf d65f03c0 (d4210000)
[ 101.542979] ---[ end trace 0000000000000000 ]---
[ 101.678650] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[ 102.046301] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001da14c98
[ 103.227334] SMP: stopping secondary CPUs
[ 103.241055] Kernel Offset: disabled
[ 103.254316] CPU features: 0x800,00184810,00001086
[ 103.268904] Memory Limit: 800 MB
[ 103.411625] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---