Re: Warnings in DRM code when removing/unbinding a driver

[John Garry]

+

So the v5.4 kernel does not have this issue.

I have bisected the initial occurrence to:

commit 37a48adfba6cf6e87df9ba8b75ab85d514ed86d8
Author: Thomas Zimmermann <tzimmermann@xxxxxxx>
Date:   Fri Sep 6 14:20:53 2019 +0200

    drm/vram: Add kmap ref-counting to GEM VRAM objects

    The kmap and kunmap operations of GEM VRAM buffers can now be called
    in interleaving pairs. The first call to drm_gem_vram_kmap() maps the
    buffer's memory to kernel address space and the final call to
    drm_gem_vram_kunmap() unmaps the memory. Intermediate calls to these
    functions increment or decrement a reference counter.

So this either exposes or creates the issue.

John

> > On Mon, 2019-12-16 at 17:23 +0000, John Garry wrote:
> > > Hi all,
> > >
> > > Enabling CONFIG_DEBUG_TEST_DRIVER_REMOVE causes many warns on a system
> > > with the HIBMC hw:
> > >
> > > [ÂÂ 27.788806] WARNING: CPU: 24 PID: 1 at
> > > drivers/gpu/drm/drm_gem_vram_helper.c:564 
> > > bo_driver_move_notify+0x8c/0x98
> >
> > A total shot in the dark. This might make no sense,
> > but it's worth a try:
>
> Thanks for the suggestion, but still the same splat.
>
> I haven't had a chance to analyze the problem myself. But perhaps we 
> should just change over the device-managed interface, as Daniel mentioned.
>
> > diff --git a/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c 
> > b/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c
> > index 2fd4ca91a62d..69bb0e29da88 100644
> > --- a/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c
> > +++ b/drivers/gpu/drm/hisilicon/hibmc/hibmc_drm_drv.c
> > @@ -247,9 +247,8 @@ static int hibmc_unload(struct drm_device *dev)
> > Â {
> > ÂÂÂÂÂÂÂÂ struct hibmc_drm_private *priv = dev->dev_private;
> > -ÂÂÂÂÂÂ hibmc_fbdev_fini(priv);
> > -
> > ÂÂÂÂÂÂÂÂ drm_atomic_helper_shutdown(dev);
> > +ÂÂÂÂÂÂ hibmc_fbdev_fini(priv);
> > ÂÂÂÂÂÂÂÂ if (dev->irq_enabled)
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ drm_irq_uninstall(dev);
> >
> > Hope it helps,
> > Ezequiel
>
> Thanks,
> John
>
> [EOM]
>
> > > [ÂÂ 27.798969] Modules linked in:
> > > [ÂÂ 27.802018] CPU: 24 PID: 1 Comm: swapper/0 Tainted: GÂÂÂ B
> > > ÂÂ 5.5.0-rc1-dirty #565
> > > [ÂÂ 27.810358] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI
> > > RC0 - V1.16.01 03/15/2019
> > > [ÂÂ 27.818872] pstate: 20c00009 (nzCv daif +PAN +UAO)
> > > [ÂÂ 27.823654] pc : bo_driver_move_notify+0x8c/0x98
> > > [ÂÂ 27.828262] lr : bo_driver_move_notify+0x40/0x98
> > > [ÂÂ 27.832868] sp : ffff00236f0677e0
> > > [ÂÂ 27.836173] x29: ffff00236f0677e0 x28: ffffa0001454e5e0
> > > [ÂÂ 27.841476] x27: ffff002366e52128 x26: ffffa000149e67b0
> > > [ÂÂ 27.846779] x25: ffff002366e523e0 x24: ffff002336936120
> > > [ÂÂ 27.852082] x23: ffff0023346f4010 x22: ffff002336936128
> > > [ÂÂ 27.857385] x21: ffffa000149c15c0 x20: ffff0023369361f8
> > > [ÂÂ 27.862687] x19: ffff002336936000 x18: 0000000000001258
> > > [ÂÂ 27.867989] x17: 0000000000001190 x16: 00000000000011d0
> > > [ÂÂ 27.873292] x15: 0000000000001348 x14: ffffa00012d68190
> > > [ÂÂ 27.878595] x13: 0000000000000006 x12: 1ffff40003241f91
> > > [ÂÂ 27.883897] x11: ffff940003241f91 x10: dfffa00000000000
> > > [ÂÂ 27.889200] x9 : ffff940003241f92 x8 : 0000000000000001
> > > [ÂÂ 27.894502] x7 : ffffa0001920fc88 x6 : ffff940003241f92
> > > [ÂÂ 27.899804] x5 : ffff940003241f92 x4 : ffff0023369363a0
> > > [ÂÂ 27.905107] x3 : ffffa00010c104b8 x2 : dfffa00000000000
> > > [ÂÂ 27.910409] x1 : 0000000000000003 x0 : 0000000000000001
> > > [ÂÂ 27.915712] Call trace:
> > > [ÂÂ 27.918151]Â bo_driver_move_notify+0x8c/0x98
> > > [ÂÂ 27.922412]Â ttm_bo_cleanup_memtype_use+0x54/0x100
> > > [ÂÂ 27.927194]Â ttm_bo_put+0x3a0/0x5d0
> > > [ÂÂ 27.930673]Â drm_gem_vram_object_free+0xc/0x18
> > > [ÂÂ 27.935109]Â drm_gem_object_free+0x34/0xd0
> > > [ÂÂ 27.939196]Â drm_gem_object_put_unlocked+0xc8/0xf0
> > > [ÂÂ 27.943978]Â hibmc_user_framebuffer_destroy+0x20/0x40
> > > [ÂÂ 27.949020]Â drm_framebuffer_free+0x48/0x58
> > > [ÂÂ 27.953194]Â drm_mode_object_put.part.1+0x90/0xe8
> > > [ÂÂ 27.957889]Â drm_mode_object_put+0x28/0x38
> > > [ÂÂ 27.961976]Â hibmc_fbdev_fini+0x54/0x78
> > > [ÂÂ 27.965802]Â hibmc_unload+0x2c/0xd0
> > > [ÂÂ 27.969281]Â hibmc_pci_remove+0x2c/0x40
> > > [ÂÂ 27.973109]Â pci_device_remove+0x6c/0x140
> > > [ÂÂ 27.977110]Â really_probe+0x174/0x548
> > > [ÂÂ 27.980763]Â driver_probe_device+0x7c/0x148
> > > [ÂÂ 27.984936]Â device_driver_attach+0x94/0xa0
> > > [ÂÂ 27.989109]Â __driver_attach+0xa8/0x110
> > > [ÂÂ 27.992935]Â bus_for_each_dev+0xe8/0x158
> > > [ÂÂ 27.996849]Â driver_attach+0x30/0x40
> > > [ÂÂ 28.000415]Â bus_add_driver+0x234/0x2f0
> > > [ÂÂ 28.004241]Â driver_register+0xbc/0x1d0
> > > [ÂÂ 28.008067]Â __pci_register_driver+0xbc/0xd0
> > > [ÂÂ 28.012329]Â hibmc_pci_driver_init+0x20/0x28
> > > [ÂÂ 28.016590]Â do_one_initcall+0xb4/0x254
> > > [ÂÂ 28.020417]Â kernel_init_freeable+0x27c/0x328
> > > [ÂÂ 28.024765]Â kernel_init+0x10/0x118
> > > [ÂÂ 28.028245]Â ret_from_fork+0x10/0x18
> > > [ÂÂ 28.031813] ---[ end trace 35a83b71b657878d ]---
> > > [ÂÂ 28.036503] ------------[ cut here ]------------
> > > [ÂÂ 28.041115] WARNING: CPU: 24 PID: 1 at
> > > drivers/gpu/drm/drm_gem_vram_helper.c:40 
> > > ttm_buffer_object_destroy+0x4c/0x80
> > > [ÂÂ 28.051537] Modules linked in:
> > > [ÂÂ 28.054585] CPU: 24 PID: 1 Comm: swapper/0 Tainted: GÂÂÂ BÂÂ W
> > > ÂÂ 5.5.0-rc1-dirty #565
> > > [ÂÂ 28.062924] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI
> > > RC0 - V1.16.01 03/15/2019
> > >
> > > [snip]
> > >
> > > Indeed, simply unbinding the device from the driver causes the same sort
> > > of issue:
> > >
> > > root@(none)$ cd ./bus/pci/drivers/hibmc-drm/
> > > root@(none)$ ls
> > > 0000:05:00.0Â bindÂÂÂÂÂÂÂÂÂ new_idÂÂÂÂÂÂÂ remove_idÂÂÂÂ uevent        
> > > unbind
> > > root@(none)$ echo 0000\:05\:00.0 > unbind
> > > [Â 116.074352] ------------[ cut here ]------------
> > > [Â 116.078978] WARNING: CPU: 17 PID: 1178 at
> > > drivers/gpu/drm/drm_gem_vram_helper.c:40 
> > > ttm_buffer_object_destroy+0x4c/0x80
> > > [Â 116.089661] Modules linked in:
> > > [Â 116.092711] CPU: 17 PID: 1178 Comm: sh Tainted: GÂÂÂ BÂÂ W
> > > 5.5.0-rc1-dirty #565
> > > [Â 116.100704] Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI
> > > RC0 - V1.16.01 03/15/2019
> > > [Â 116.109218] pstate: 20400009 (nzCv daif +PAN -UAO)
> > > [Â 116.114001] pc : ttm_buffer_object_destroy+0x4c/0x80
> > > [Â 116.118956] lr : ttm_buffer_object_destroy+0x18/0x80
> > > [Â 116.123910] sp : ffff0022e6cef8e0
> > > [Â 116.127215] x29: ffff0022e6cef8e0 x28: ffff00231b1fb000
> > > [Â 116.132519] x27: 0000000000000000 x26: ffff00231b1fb000
> > > [Â 116.137821] x25: ffff0022e6cefdc0 x24: 0000000000002480
> > > [Â 116.143124] x23: ffff0023682b6ab0 x22: ffff0023682b6800
> > > [Â 116.148427] x21: ffff0023682b6800 x20: 0000000000000000
> > > [Â 116.153730] x19: ffff0023682b6800 x18: 0000000000000000
> > > [Â 116.159032] x17: 000000000000000000000000001
> > > [Â 116.185545] x7 : ffff0023682b6b07 x6 : ffff80046d056d61
> > > [Â 116.190848] x5 : ffff80046d056d61 x4 : ffff0023682b6ba0
> > > [Â 116.196151] x3 : ffffa00010197338 x2 : dfffa00000000000
> > > [Â 116.201453] x1 : 0000000000000003 x0 : 0000000000000001
> > > [Â 116.206756] Call trace:
> > > [Â 116.209195]Â ttm_buffer_object_destroy+0x4c/0x80
> > > [Â 116.213803]Â ttm_bo_release_list+0x184/0x220
> > > [Â 116.218064]Â ttm_bo_put+0x410/0x5d0
> > > [Â 116.221544]Â drm_gem_vram_object_free+0xc/0x18
> > > [Â 116.225979]Â drm_gem_object_free+0x34/0xd0
> > > [Â 116.230066]Â drm_gem_object_put_unlocked+0xc8/0xf0
> > > [Â 116.234848]Â hibmc_user_framebuffer_destroy+0x20/0x40
> > > [Â 116.239890]Â drm_framebuffer_free+0x48/0x58
> > > [Â 116.244064]Â drm_mode_object_put.part.1+0x90/0xe8
> > > [Â 116.248759]Â drm_mode_object_put+0x28/0x38
> > > [Â 116.252846]Â hibmc_fbdev_fini+0x54/0x78
> > > [Â 116.256672]Â hibmc_unload+0x2c/0xd0
> > > [Â 116.260151]Â hibmc_pci_remove+0x2c/0x40
> > > [Â 116.263979]Â pci_device_remove+0x6c/0x140
> > > [Â 116.267980]Â device_release_driver_internal+0x134/0x250
> > > [Â 116.273196]Â device_driver_detach+0x28/0x38
> > > [Â 116.277369]Â unbind_store+0xfc/0x150
> > > [Â 116.280934]Â drv_attr_store+0x48/0x60
> > > [Â 116.284589]Â sysfs_kf_write+0x80/0xb0
> > > [Â 116.288241]Â kernfs_fop_write+0x1d4/0x320
> > > [Â 116.292243]Â __vfs_write+0x54/0x98
> > > [Â 116.295635]Â vfs_write+0xe8/0x270
> > > [Â 116.298940]Â ksys_write+0xc8/0x180
> > > [Â 116.302333]Â __arm64_sys_write+0x40/0x50
> > > [Â 116.306248]Â el0_svc_common.constprop.0+0xa4/0x1f8
> > > [Â 116.311029]Â el0_svc_handler+0x34/0xb0
> > > [Â 116.314770]Â el0_sync_handler+0x10c/0x1c8
> > > [Â 116.318769]Â el0_sync+0x140/0x180
> > > [Â 116.322074] ---[ end trace e60e43d0e316b5c8 ]---
> > > [Â 116.326868] ------------[ cut here ]------------
> > >
> > >
> > > dmesg and .config is here:
> > > https://pastebin.com/4P5yaZBS
> > >
> > > I'm not sure if this is a HIBMC driver issue or issue with the 
> > > framework.
> > >
> > > john
> > >
> > >
> > > _______________________________________________
> > > dri-devel mailing list
> > > dri-devel@xxxxxxxxxxxxxxxxxxxxx
> > > https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
> _______________________________________________
> Linuxarm mailing list
> Linuxarm@xxxxxxxxxx
> http://hulk.huawei.com/mailman/listinfo/linuxarm
> .