[PATCH] drm/modes: reject out-of-range GTF blanking duty cycle

[Xiang Mei]

drm_gtf_mode_complex() derives the blanking duty cycle from the GTF
C/M/K/J coefficients and divides by its complement:

	ideal_duty_cycle = GTF_C_PRIME * 1000 -
				(GTF_M_PRIME * 1000000 / hfreq_est);
	hblank = total_active_pixels * ideal_duty_cycle /
			(100000 - ideal_duty_cycle);

ideal_duty_cycle is an unsigned per-mille percentage that must stay below
100000, but nothing enforces it. Via drm_gtf2_mode() the coefficients come
straight from an EDID secondary-GTF range descriptor, so a crafted monitor
(or an edid_override re-probed with DRM_IOCTL_MODE_GETCONNECTOR) can set
ideal_duty_cycle to exactly 100000, making the divisor 0 and trapping with
a divide error; a larger value wraps the unsigned subtraction and yields a
garbage modeline.

Reject a duty cycle of 100% or more, which is never a valid GTF result.
drm_gtf2_mode() and the inferred-mode helpers already handle a NULL return.

Crash:

  Oops: divide error: 0000 [#1] SMP KASAN NOPTI
  RIP: 0010:drm_gtf_mode_complex (drivers/gpu/drm/drm_modes.c:972)
  Call Trace:
   drm_gtf2_mode (drivers/gpu/drm/drm_edid.c:3365)
   drm_mode_std (drivers/gpu/drm/drm_edid.c:3464)
   _drm_edid_connector_add_modes.part.0 (drivers/gpu/drm/drm_edid.c:4024)
   drm_edid_connector_add_modes (drivers/gpu/drm/drm_edid.c:7020)
   bochs_connector_helper_get_modes (drivers/gpu/drm/tiny/bochs.c:575)
   drm_helper_probe_single_connector_modes (drivers/gpu/drm/drm_probe_helper.c:426)
   drm_mode_getconnector (drivers/gpu/drm/drm_connector.c:3376)
   ...
   __x64_sys_ioctl (fs/ioctl.c:597)
   do_syscall_64 (arch/x86/entry/syscall_64.c:94)
   entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121)
  Kernel panic - not syncing: Fatal exception

Fixes: 26bbdadad356 ("drm/mode: add the GTF algorithm in kernel space")
Reported-by: Weiming Shi <bestswngs@xxxxxxxxx>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Xiang Mei <xmei5@xxxxxxx>
---
 drivers/gpu/drm/drm_modes.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/drm_modes.c b/drivers/gpu/drm/drm_modes.c
index 3f8e025fd6d9..b6f263e9ba86 100644
--- a/drivers/gpu/drm/drm_modes.c
+++ b/drivers/gpu/drm/drm_modes.c
@@ -966,6 +966,10 @@ drm_gtf_mode_complex(struct drm_device *dev, int hdisplay, int vdisplay,
 	/* 18.Find the ideal blanking duty cycle from blanking duty cycle */
 	ideal_duty_cycle = GTF_C_PRIME * 1000 -
 				(GTF_M_PRIME * 1000000 / hfreq_est);
+	if (ideal_duty_cycle >= 100000) {
+		drm_mode_destroy(dev, drm_mode);
+		return NULL;
+	}
 	/* 19.Find the number of pixels in the blanking time to the nearest
 	 * double character cell: */
 	hblank = total_active_pixels * ideal_duty_cycle /
-- 
2.43.0