[PATCH v1 01/11] media: intel/ipu6: move busy check to ipu6_isys_vidioc_s_fmt_vid_cap()

[Valerii Sokolov]

ipu6_isys_vidioc_try_fmt_vid_cap() rejected calls with -EBUSY when
buffers were allocated on the queue (vb2_is_busy()), although the V4L2
specification states that VIDIOC_TRY_FMT can be called at any time and
never returns EBUSY.

Conversely, ipu6_isys_vidioc_s_fmt_vid_cap() lacked the vb2_is_busy()
check (unlike ipu6_isys_vidioc_s_fmt_meta_cap()) and ignored the return
value of ipu6_isys_vidioc_try_fmt_vid_cap(). As a result, when a client
issued VIDIOC_S_FMT while the queue was busy, try_fmt returned -EBUSY
before clamping fields or computing bytesperline and sizeimage, and
s_fmt copied the unvalidated user format into av->pix_fmt.

Corrupting av->pix_fmt while buffers are allocated breaks subsequent
buffer preparation in ipu6_isys_buf_prepare(): if user-space left
sizeimage as 0, the size > vb2_plane_size(vb, 0) check is bypassed and a
mismatched bytesperline * height passed to vb2_set_plane_payload() can
trigger WARN_ON_ONCE(size > vb->planes[plane_no].length), or if
VIDIOC_STREAMON follows, program the firmware to DMA a larger frame into
smaller buffers.

Move the vb2_is_busy() check from ipu6_isys_vidioc_try_fmt_vid_cap() to
ipu6_isys_vidioc_s_fmt_vid_cap(), matching the meta capture callbacks.

Fixes: 3c1dfb5a69cf ("media: intel/ipu6: input system video nodes and buffer queues")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Valerii Sokolov <valerii.sokolov@xxxxx>
---
 drivers/media/pci/intel/ipu6/ipu6-isys-video.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/media/pci/intel/ipu6/ipu6-isys-video.c b/drivers/media/pci/intel/ipu6/ipu6-isys-video.c
index 129016e574..f394e78db6 100644
--- a/drivers/media/pci/intel/ipu6/ipu6-isys-video.c
+++ b/drivers/media/pci/intel/ipu6/ipu6-isys-video.c
@@ -290,9 +290,6 @@ static int ipu6_isys_vidioc_try_fmt_vid_cap(struct file *file, void *fh,
 {
 	struct ipu6_isys_video *av = video_drvdata(file);
 
-	if (vb2_is_busy(&av->aq.vbq))
-		return -EBUSY;
-
 	__ipu6_isys_vidioc_try_fmt_vid_cap(av, f);
 
 	return 0;
@@ -324,6 +321,9 @@ static int ipu6_isys_vidioc_s_fmt_vid_cap(struct file *file, void *fh,
 {
 	struct ipu6_isys_video *av = video_drvdata(file);
 
+	if (vb2_is_busy(&av->aq.vbq))
+		return -EBUSY;
+
 	ipu6_isys_vidioc_try_fmt_vid_cap(file, fh, f);
 	av->pix_fmt = f->fmt.pix;
 
-- 
2.56.0