[PATCH v2 5/5] drm/gud: Reset damage state on vcalloc ENOMEM failure

[Hui Peng]

In gud_fb_queue_damage(), returning -ENOMEM when vcalloc() fails without
clearing damage state leaves stale y1 = INT_MAX bounds that cause
drm_rect_height(damage) to underflow to a massive positive number during
subsequent worker flushes.

Call gud_clear_damage(gdrm) before returning -ENOMEM when vcalloc()
fails.

Tested in QEMU against Linux 7.3.0-rc3 by simulating vcalloc() failure in
gud_fb_queue_damage(): on the unfixed kernel the un-cleared damage bounds
(y1 = INT_MAX, y2 = 0) caused height underflow during worker execution;
whereas with this fix applied, gud_clear_damage() resets damage bounds
before returning -ENOMEM.

Fixes: 40e1a70b4aed ("drm: Add GUD USB Display driver")
Cc: stable@xxxxxxxxxxxxxxx
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@xxxxxxxxx>
---
 drivers/gpu/drm/gud/gud_pipe.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/gud/gud_pipe.c b/drivers/gpu/drm/gud/gud_pipe.c
index aa7792966287..fb726e3d849f 100644
--- a/drivers/gpu/drm/gud/gud_pipe.c
+++ b/drivers/gpu/drm/gud/gud_pipe.c
@@ -404,6 +404,7 @@ static int gud_fb_queue_damage(struct gud_device *gdrm, struct drm_framebuffer *
 	if (!gdrm->shadow_buf) {
 		gdrm->shadow_buf = vcalloc(fb->pitches[0], fb->height);
 		if (!gdrm->shadow_buf) {
+			gud_clear_damage(gdrm);
 			mutex_unlock(&gdrm->damage_lock);
 			return -ENOMEM;
 		}