[BUG] drm/vmwgfx: vmw_cmdbuf_alloc leaks ~103 MiB/day via fbdev damage worker

[Дима Дьячков]

Hi,

I have a reproducible, quantified kernel memory leak in vmwgfx on a
headless VMware guest. It is attributed to a specific call site with
slab_debug, and I have a clean causal test that stops it. I could not
find this reported anywhere - I searched the dri-devel archives, the
mainline and 6.12.y git logs, and the distro trackers.

Summary
=======

On a headless VMware guest with the text console bound to the vmwgfx
framebuffer, the fbdev-emulation damage worker issues a full atomic
commit for every console damage event, and vmwgfx leaks a command
buffer on each one. The result is unreclaimable kernel slab growing
linearly at ~103 MiB/day, never reclaimed under memory pressure,
released only by reboot.

Environment
===========

Kernel: 6.12.100+deb13-amd64 (Debian 13 stock, tracks 6.12.y)
Guest: VMware Virtual Platform, BIOS 6.00
Console: headless, over SSH; a single agetty on the console
fb0: vmwgfxdrmfb
vtcon0: "(M) frame buffer device", bind=1
Config: CONFIG_HZ=250, CONFIG_SLUB_DEBUG=y, CONFIG_STACKDEPOT=y
(CONFIG_DEBUG_KMEMLEAK is not set in Debian's kernel)

i915 is also loaded on this host (a passed-through GPU used for video
transcoding) but is unrelated - the console framebuffer is vmwgfx.

Symptom
=======

SUnreclaim grows linearly at ~103 MiB/day and is never reclaimed under
pressure. The growth is confined to the plain kmalloc-256 cache -
notably *not* kmalloc-cg-256, so it is plain GFP_KERNEL from an
in-kernel allocator rather than anything cgroup-accounted.

Attribution
===========

Booted with slab_debug=U,kmalloc-256, then read
/sys/kernel/debug/slab/kmalloc-256/alloc_traces, which ranks by live
object count. Top entry (header line wrapped by hand for width):

41025 vmw_cmdbuf_alloc+0x40/0x200 [vmwgfx] waste=328200/8
age=254700/1301825/2351179 pid=9-208926 cpus=0-5
vmw_cmdbuf_alloc+0x40/0x200 [vmwgfx]
vmw_cmdbuf_reserve+0x142/0x170 [vmwgfx]
vmw_cmd_ctx_reserve+0x24/0x60 [vmwgfx]
vmw_stdu_bind_st+0x6b/0x100 [vmwgfx]
vmw_stdu_primary_plane_atomic_update+0xa8/0x1d0 [vmwgfx]
drm_atomic_helper_commit_planes+0x112/0x320 [drm_kms_helper]
drm_atomic_helper_commit_tail+0x48/0x90 [drm_kms_helper]
vmw_atomic_commit_tail+0x1b/0x80 [vmwgfx]
commit_tail+0x91/0x130 [drm_kms_helper]
drm_atomic_helper_commit+0x11a/0x140 [drm_kms_helper]
drm_atomic_commit+0xa9/0xe0 [drm]
drm_atomic_helper_dirtyfb+0x1c6/0x2a0 [drm_kms_helper]
drm_fbdev_ttm_helper_fb_dirty+0x216/0x310 [drm_ttm_helper]
drm_fb_helper_damage_work+0x94/0x170 [drm_kms_helper]
process_one_work+0x174/0x330
worker_thread+0x191/0x2f0

The next-largest entry in that cache held 1056 objects - this one is
~30x everything else combined.

Quantitative confirmation
=========================

With CONFIG_HZ=250, from the age=min/avg/max field above:

- max age 2,351,179 jiffies = 9405 s = full uptime at time of read
- avg age is almost exactly half of max, which is the signature of
uniform allocation with zero frees
- 41,025 objects over the 8386 s in which allocation was active
= 4.89 objects/sec = 4.30 MiB/h = ~103 MiB/day

That matches the independently measured SUnreclaim growth to within
noise.

Causal test
===========

Unbinding the text console from the framebuffer stops it instantly:

# echo 0 > /sys/class/vtconsole/vtcon0/bind

After that the minimum object age jumped from 26 jiffies to 254,700
and kept climbing - i.e. no new allocations at all. SUnreclaim also
dropped ~93 MiB immediately as the fbdev shadow buffer was released,
and has fallen steadily since.

Two things I checked so you don't have to
=========================================

This is not a recent regression. vmwgfx_cmdbuf.c has had no functional
change between v6.12 and current mainline - only the command buffer ID
addition (922f9de) and the treewide kmalloc -> kmalloc_obj conversions.
vmwgfx_stdu.c is similarly untouched in the relevant paths, and nothing
in the 6.12.y stable series touches cmdbuf, damage or dirtyfb handling.
So this looks long-standing rather than newly introduced.

It is also not caused by the fbdev-ttm conversion, which is the obvious
suspect given the stack. drm_fbdev_ttm was a pure rename of
drm_fbdev_generic (aae4682e5d66, v6.10), and vmwgfx has used the
generic DRM fbdev emulation since v6.2 (df42523c12f8, which deleted
vmwgfx_fb.c). What did change at v6.2 is that console damage now goes
through a full atomic commit, which the old vmwgfx_fb.c did not do -
but the leak itself is on the vmwgfx side.

Possibly related
================

There is an open, unresolved report of a different vmwgfx leak in the
same atomic-commit path - Yuma Kakei, 12 Mar 2026, "[BUG] vmwgfx:
Severe Slab memory leak (radix_tree_node) with Wayland Atomic KMS"
(kernels 6.17 / 7.0-rc3, KWin Wayland with Atomic KMS, stops with
KWIN_DRM_NO_AMS=1). Ian Forbes proposed a patch; the reporter confirmed
on 15 Mar 2026 that it did not fix it.

I do *not* claim these are the same bug - different slab caches,
different trigger, different kernels. But both are vmwgfx leaking on
every atomic commit and stopping dead when the atomic path stops, so
they may share a root cause.

Workarounds
===========

- echo 0 > /sys/class/vtconsole/vtcon0/bind
runtime, reversible, stops it immediately

- drm_kms_helper.fbdev_emulation=0 on the kernel command line
cleaner for a headless host; disables fbdev emulation outright so
no damage worker exists. Must be set at boot.

- modprobe.blacklist=vmwgfx
heavy-handed; loses all console output

All three cost the virtual console, which is fine headless but is not
a general fix.

Happy to test patches, run instrumented kernels, or provide further
data - the system reproduces this continuously and I can rebind the
console at will to toggle it on and off.

Thanks,