[PATCH 2/2] drm/radeon: fix integer overflow in radeon_align_pitch()

[Werner Kasselman]

radeon_align_pitch() has the same integer overflow as amdgpu's variant:
'aligned * cpp' can overflow signed int to 0 when alignment rounding
pushes the width past INT_MAX/cpp. This produces a 0-byte GEM buffer
via radeon_mode_dumb_create(), reachable from unprivileged userspace
via DRM_IOCTL_MODE_CREATE_DUMB on the render node.

Add an overflow check in radeon_align_pitch() and reject zero pitch/size
in radeon_mode_dumb_create().

Found via AST-based call-graph analysis using sqry.

Fixes: ff72145badb8 ("drm: dumb scanout create/mmap for intel/radeon (v3)")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Werner Kasselman <werner@xxxxxxxxxxx>
---
 drivers/gpu/drm/radeon/radeon_gem.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/gpu/drm/radeon/radeon_gem.c b/drivers/gpu/drm/radeon/radeon_gem.c
index 20fc87409f2e..2cd179fef347 100644
--- a/drivers/gpu/drm/radeon/radeon_gem.c
+++ b/drivers/gpu/drm/radeon/radeon_gem.c
@@ -828,6 +828,11 @@ int radeon_align_pitch(struct radeon_device *rdev, int width, int cpp, bool tile
 
 	aligned += pitch_mask;
 	aligned &= ~pitch_mask;
+
+	/* Guard against integer overflow in aligned * cpp. */
+	if (aligned > INT_MAX / (cpp ? cpp : 1) || aligned <= 0)
+		return 0;
+
 	return aligned * cpp;
 }
 
@@ -842,8 +847,12 @@ int radeon_mode_dumb_create(struct drm_file *file_priv,
 
 	args->pitch = radeon_align_pitch(rdev, args->width,
 					 DIV_ROUND_UP(args->bpp, 8), 0);
+	if (!args->pitch)
+		return -EINVAL;
 	args->size = (u64)args->pitch * args->height;
 	args->size = ALIGN(args->size, PAGE_SIZE);
+	if (!args->size)
+		return -EINVAL;
 
 	r = radeon_gem_object_create(rdev, args->size, 0,
 				     RADEON_GEM_DOMAIN_VRAM, 0,
-- 
2.43.0