[PATCH 3.12 67/98] saa7134: Fix bytesperline not being set correctly for planar formats

From: Jiri Slaby
Date: Mon Apr 11 2016 - 09:41:25 EST


From: Hans de Goede <hdegoede@xxxxxxxxxx>

3.12-stable review patch. If anyone has any objections, please let me know.

===============

commit 3e71da19f9dc22e39a755d6ae9678661abb66adc upstream.

bytesperline should be the bytesperline for the first plane for planar
formats, not that of all planes combined.

This fixes a crash in xawtv caused by the wrong bpl.

BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1305389
Reported-and-tested-by: Stas Sergeev <stsp@xxxxxxx>

Signed-off-by: Hans de Goede <hdegoede@xxxxxxxxxx>
Signed-off-by: Hans Verkuil <hans.verkuil@xxxxxxxxx>
Signed-off-by: Mauro Carvalho Chehab <mchehab@xxxxxxxxxxxxxxx>
Signed-off-by: Jiri Slaby <jslaby@xxxxxxx>
---
drivers/media/pci/saa7134/saa7134-video.c | 18 ++++++++++++------
1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c
index fb60da85bc2c..d11fdfbac275 100644
--- a/drivers/media/pci/saa7134/saa7134-video.c
+++ b/drivers/media/pci/saa7134/saa7134-video.c
@@ -1562,10 +1562,13 @@ static int saa7134_g_fmt_vid_cap(struct file *file, void *priv,
f->fmt.pix.height = dev->height;
f->fmt.pix.field = fh->cap.field;
f->fmt.pix.pixelformat = dev->fmt->fourcc;
- f->fmt.pix.bytesperline =
- (f->fmt.pix.width * dev->fmt->depth) >> 3;
+ if (dev->fmt->planar)
+ f->fmt.pix.bytesperline = f->fmt.pix.width;
+ else
+ f->fmt.pix.bytesperline =
+ (f->fmt.pix.width * dev->fmt->depth) / 8;
f->fmt.pix.sizeimage =
- f->fmt.pix.height * f->fmt.pix.bytesperline;
+ (f->fmt.pix.height * f->fmt.pix.width * dev->fmt->depth) / 8;
f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M;
f->fmt.pix.priv = 0;
return 0;
@@ -1646,10 +1649,13 @@ static int saa7134_try_fmt_vid_cap(struct file *file, void *priv,
if (f->fmt.pix.height > maxh)
f->fmt.pix.height = maxh;
f->fmt.pix.width &= ~0x03;
- f->fmt.pix.bytesperline =
- (f->fmt.pix.width * fmt->depth) >> 3;
+ if (fmt->planar)
+ f->fmt.pix.bytesperline = f->fmt.pix.width;
+ else
+ f->fmt.pix.bytesperline =
+ (f->fmt.pix.width * fmt->depth) / 8;
f->fmt.pix.sizeimage =
- f->fmt.pix.height * f->fmt.pix.bytesperline;
+ (f->fmt.pix.height * f->fmt.pix.width * fmt->depth) / 8;
f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M;
f->fmt.pix.priv = 0;

--
2.8.1