Re: [PATCH v4 0/8] Add MT8173 Video Encoder Driver and VPU Driver

[tiffany lin]

Hi Hans,

On Mon, 2016-02-15 at 13:04 +0100, Hans Verkuil wrote:
> Hi Tiffany,
> 
> Once again, thanks for your patch series!
> 
> I have some more comments below.
> 
> On 02/04/2016 12:34 PM, Tiffany Lin wrote:
> > 
> > ==============
> >  Introduction
> > ==============
> > 
> > The purpose of this series is to add the driver for video codec hw embedded in the Mediatek's MT8173 SoCs.
> > Mediatek Video Codec is able to handle video encoding of in a range of formats.
> > 
> > This patch series also include VPU driver. Mediatek Video Codec driver rely on VPU driver to load, communicate with VPU.
> > 
> > Internally the driver uses videobuf2 framework and MTK IOMMU and MTK SMI.
> > MTK IOMMU[1] and MTK SMI[2] have not yet been merged, but we wanted to start discussion about the driver earlier so it could be merged sooner.
> > 
> > [1]https://patchwork.kernel.org/patch/8120921/
> > [2]https://patchwork.kernel.org/patch/8120941/
> > 
> > 
> > ==================
> >  Device interface
> > ==================
> > 
> > In principle the driver bases on v4l2 memory-to-memory framework:
> > it provides a single video node and each opened file handle gets its own private context with separate buffer queues. Each context consist of 2 buffer queues: OUTPUT (for source buffers, i.e. raw video frames) and CAPTURE (for destination buffers, i.e. encoded video frames).
> > 
> > ==============================
> >  VPU (Video Processor Unit)
> > ==============================
> > The VPU driver for hw video codec embedded in Mediatek's MT8173 SOCs.
> > It is able to handle video decoding/encoding in a range of formats.
> > The driver provides with VPU firmware download, memory management and the communication interface between CPU and VPU.
> > For VPU initialization, it will create virtual memory for CPU access and physical address for VPU hw device access. When a decode/encode instance opens a device node, vpu driver will download vpu firmware to the device.
> > A decode/encode instant will decode/encode a frame using VPU interface to interrupt vpu to handle decoding/encoding jobs.
> > 
> > Please have a look at the code and comments will be very much appreciated.
> > 
> > 
> > Change in v4:
> > Vcodec Part
> > 1. Remove MTK_ENCODE_PARAM_SKIP_FRAME support
> > 2. Remove MTK_ENCODE_PARAM_FRAME_TYPE and change to use V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME[3]
> > 3. Refine Encoder HW clock source
> > 4. Refine debug log
> > 5. Add watchdog support
> > 6. With patch "media: v4l2-compat-ioctl32: fix missing length copy in put_v4l2_buffer32"[4], v4l2-compliance test passed[5] in v4.4-rc5 
> > 
> > VPU Part
> > 1. These two patches were Acked-by: Rob Herring <robh@xxxxxxxxxx> in v3
> >    [PATCH v3 1/8] dt-bindings: Add a binding for Mediatek Video Processor
> >    [PATCH v3 3/8] arm64: dts: mediatek: Add node for Mediatek Video Processor Unit
> >    Because we were wrong about how the hardware works, there is no connection between VPU and IOMMU HW
> >    We remove VPU attaching to IOMMU
> > 2. Support VPU running on 4GB DRAM system
> > 3. Support VPU watchdog reset
> > 4. Refine for coding style 
> > 
> > [3]https://patchwork.linuxtv.org/patch/32670/
> > [4] https://patchwork.linuxtv.org/patch/32631/
> > [5]localhost ~ # /usr/bin/v4l2-compliance -d /dev/video1
> > Driver Info:
> >         Driver name   : mtk-vcodec-en
> >         Card type     : platform:mt817
> >         Bus info      : platform:mt817
> 
> Perhaps this should be mt817x?
> 
> >         Driver version: 4.4.0
> >         Capabilities  : 0x84204000
> >                 Video Memory-to-Memory Multiplanar
> >                 Streaming
> >                 Extended Pix Format
> >                 Device Capabilities
> >         Device Caps   : 0x04204000
> >                 Video Memory-to-Memory Multiplanar
> >                 Streaming
> >                 Extended Pix Format
> > 
> > Compliance test for device /dev/video1 (not using libv4l2):
> > 
> > Required ioctls:
> >         test VIDIOC_QUERYCAP: OK
> > 
> > Allow for multiple opens:
> >         test second video open: OK
> >         test VIDIOC_QUERYCAP: OK
> >         test VIDIOC_G/S_PRIORITY: OK
> > 
> > Debug ioctls:
> >         test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> >         test VIDIOC_LOG_STATUS: OK (Not Supported)
> > 
> > Input ioctls:
> >         test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> >         test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >         test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> >         test VIDIOC_ENUMAUDIO: OK (Not Supported)
> >         test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> >         test VIDIOC_G/S_AUDIO: OK (Not Supported)
> >         Inputs: 0 Audio Inputs: 0 Tuners: 0
> > 
> > Output ioctls:
> >         test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> >         test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >         test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> >         test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> >         test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> >         Outputs: 0 Audio Outputs: 0 Modulators: 0
> > 
> > Input/Output configuration ioctls:
> >         test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> >         test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> >         test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> >         test VIDIOC_G/S_EDID: OK (Not Supported)
> > 
> >         Control ioctls:
> >                 test VIDIOC_QUERYCTRL/MENU: OK
> >                 test VIDIOC_G/S_CTRL: OK
> >                 test VIDIOC_G/S/TRY_EXT_CTRLS: OK
> >                 test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
> >                 test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> >                 Standard Controls: 12 Private Controls: 0
> > 
> >         Format ioctls:
> >                 test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> >                 test VIDIOC_G/S_PARM: OK (Not Supported)
> >                 test VIDIOC_G_FBUF: OK (Not Supported)
> >                 test VIDIOC_G_FMT: OK
> >                 test VIDIOC_TRY_FMT: OK
> >                 test VIDIOC_S_FMT: OK
> >                 test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> > 
> >         Codec ioctls:
> >                 test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> >                 test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> >                 test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
> > 
> >         Buffer ioctls:
> >                 warn: ../../../v4l-utils-1.6.0/utils/v4l2-compliance/v4l2-test-buffers.cpp(475): VIDIOC_CREATE_BUFS not supported
> >                 warn: ../../../v4l-utils-1.6.0/utils/v4l2-compliance/v4l2-test-buffers.cpp(475): VIDIOC_CREATE_BUFS not supported
> >                 test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> >                 test VIDIOC_EXPBUF: OK
> > 
> > Total: 38, Succeeded: 38, Failed: 0, Warnings: 2
> 
> Looks great!
> 
> Having reviewed the v4 patch series I'd say it is close to being ready (at least
> w.r.t. the V4L2 parts). Most of my comments are minor, but fixing the open()
> limitation is probably a bigger change on your side.
> 
> It should be well worth the effort, though. In my experience limiting the number
> of time you can call open() is always a sign of a design flaw and the code improves
> if this is handled in the right place (usually start_streaming, which is when you
> actually start to use the hardware).
> 
> And besides, limiting the number of open()s is out-of-spec.
> 
You are right. We did not think about this before.
We will remove open() limitation and put limitation control in right
place.
Appreciated your time and suggestions.



> Regards,
> 
> 	Hans
> 
> > 
> > 
> > Change in v3:
> > 1.Refine code to pass v4l2-compliance test, now it still has 2 issues 2.Refine code according to latest MTK IOMMU patches[1] 3.Remove MFC51 specific CIDs and add MTK specific CIDs for for keyframe and
> >   skip I-frame
> > 4.Refine code according to review comments
> > 
> > Below is the v1.6 version v4l2-compliance report for the mt8173 encoder driver.
> > Now there are still 2 test fail in v1.6.
> > For VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF, we directly use v4l2_m2m_ioctl_* functions, but it still fail. It pass in kernel 3.18 but fail in kernel 4.4.
> > We will try v1.8 in next version.
> > VIDIOC_EXPBUF is becuase we support all three memory types VB2_DMABUF, VB2_MMAP and VB2_USERPTR. VIDIOC_EXPBUF only allowed when only VB2_MMAP supported.
> > localhost ~ # /usr/bin/v4l2-compliance -d /dev/video1 Driver Info:
> >         Driver name   : mtk-vcodec-en
> >         Card type     : platform:mt817
> >         Bus info      : platform:mt817
> >         Driver version: 4.4.0
> >         Capabilities  : 0x84204000
> >                 Video Memory-to-Memory Multiplanar
> >                 Streaming
> >                 Extended Pix Format
> >                 Device Capabilities
> >         Device Caps   : 0x04204000
> >                 Video Memory-to-Memory Multiplanar
> >                 Streaming
> >                 Extended Pix Format
> > 
> > Compliance test for device /dev/video1 (not using libv4l2):
> > 
> > Required ioctls:
> >         test VIDIOC_QUERYCAP: OK
> > 
> > Allow for multiple opens:
> >         test second video open: OK
> >         test VIDIOC_QUERYCAP: OK
> >         test VIDIOC_G/S_PRIORITY: OK
> > 
> > Debug ioctls:
> >         test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> >         test VIDIOC_LOG_STATUS: OK (Not Supported)
> > 
> > Input ioctls:
> >         test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> >         test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >         test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> >         test VIDIOC_ENUMAUDIO: OK (Not Supported)
> >         test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> >         test VIDIOC_G/S_AUDIO: OK (Not Supported)
> >         Inputs: 0 Audio Inputs: 0 Tuners: 0
> > 
> > Output ioctls:
> >         test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> >         test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> >         test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> >         test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> >         test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> >         Outputs: 0 Audio Outputs: 0 Modulators: 0
> > 
> > Input/Output configuration ioctls:
> >         test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> >         test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> >         test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> >         test VIDIOC_G/S_EDID: OK (Not Supported)
> > 
> >         Control ioctls:
> >                 test VIDIOC_QUERYCTRL/MENU: OK
> >                 test VIDIOC_G/S_CTRL: OK
> >                 test VIDIOC_G/S/TRY_EXT_CTRLS: OK
> >                 test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
> >                 test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> >                 Standard Controls: 11 Private Controls: 2
> > 
> >         Format ioctls:
> >                 test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> >                 test VIDIOC_G/S_PARM: OK (Not Supported)
> >                 test VIDIOC_G_FBUF: OK (Not Supported)
> >                 test VIDIOC_G_FMT: OK
> >                 test VIDIOC_TRY_FMT: OK
> >                 test VIDIOC_S_FMT: OK
> >                 test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> > 
> >         Codec ioctls:
> >                 test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> >                 test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> >                 test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
> > 
> >         Buffer ioctls:
> >                 fail: ../../../v4l-utils-1.6.0/utils/v4l2-compliance/v4l2-test-buffers.cpp(266): vp->length == 0
> >                 fail: ../../../v4l-utils-1.6.0/utils/v4l2-compliance/v4l2-test-buffers.cpp(335): buf.check(Unqueued, i)
> >                 fail: ../../../v4l-utils-1.6.0/utils/v4l2-compliance/v4l2-test-buffers.cpp(420): testQueryBuf(node, i, q.g_buffers())
> >                 test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
> >                 fail: ../../../v4l-utils-1.6.0/utils/v4l2-compliance/v4l2-test-buffers.cpp(500): q.has_expbuf(node)
> >                 test VIDIOC_EXPBUF: FAIL
> > Total: 38, Succeeded: 36, Failed: 2, Warnings: 0
> > 
> > 
> > Change in v2:
> > Vcodec Part
> > 1.Remove common and include directory in mtk-vcodec 2.Refine vb2ops_venc_start_streaming and vb2ops_venc_stop_streaming and state machine 3.Remove venc_if_init and venc_if_deinit 4.Refine debug message 5.Refine lab and vpu decription in mediatek-vcodec.txt
> > 
> > VPU Part
> > 1. Modify VPU Kconfig
> > 2. Move encoder header files to other patch sets 3. Remove marcos for extended virtual/iova address 4. Change register and variable names 5. Add a reference counter for VPU watchdog 6. Remove one busy waiting in function vpu_ipi_send 7. Operate VPU clock in VPU driver (not called by encoder drivers) 8. Refine memory mapping, firmware download and extended memory allocation/free functions 9. Release more allocated resources in driver remove function
> > 
> > 
> > 
> > Andrew-CT Chen (2):
> >   dt-bindings: Add a binding for Mediatek Video Processor
> >   [Media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver
> > 
> > Tiffany Lin (6):
> >   [media] VPU: mediatek: support Mediatek VPU
> >   arm64: dts: mediatek: Add node for Mediatek Video Processor Unit
> >   dt-bindings: Add a binding for Mediatek Video Encoder
> >   [media] vcodec: mediatek: Add Mediatek VP8 Video Encoder Driver
> >   [media] vcodec: mediatek: Add Mediatek H264 Video Encoder Driver
> >   arm64: dts: mediatek: Add Video Encoder for MT8173
> > 
> >  .../devicetree/bindings/media/mediatek-vcodec.txt  |   59 +
> >  .../devicetree/bindings/media/mediatek-vpu.txt     |   31 +
> >  arch/arm64/boot/dts/mediatek/mt8173.dtsi           |   62 +
> >  drivers/media/platform/Kconfig                     |   20 +
> >  drivers/media/platform/Makefile                    |    4 +
> >  drivers/media/platform/mtk-vcodec/Makefile         |   11 +
> >  .../media/platform/mtk-vcodec/h264_enc/Makefile    |    6 +
> >  .../platform/mtk-vcodec/h264_enc/venc_h264_if.c    |  540 ++++++++
> >  .../platform/mtk-vcodec/h264_enc/venc_h264_if.h    |  165 +++
> >  .../platform/mtk-vcodec/h264_enc/venc_h264_vpu.c   |  309 +++++
> >  .../platform/mtk-vcodec/h264_enc/venc_h264_vpu.h   |   30 +
> >  drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h |  388 ++++++
> >  drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1380 ++++++++++++++++++++
> >  drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.h |   46 +
> >  .../media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c |  476 +++++++
> >  .../media/platform/mtk-vcodec/mtk_vcodec_enc_pm.c  |  132 ++
> >  .../media/platform/mtk-vcodec/mtk_vcodec_intr.c    |  102 ++
> >  .../media/platform/mtk-vcodec/mtk_vcodec_intr.h    |   29 +
> >  drivers/media/platform/mtk-vcodec/mtk_vcodec_pm.h  |   26 +
> >  .../media/platform/mtk-vcodec/mtk_vcodec_util.c    |  106 ++
> >  .../media/platform/mtk-vcodec/mtk_vcodec_util.h    |   85 ++
> >  drivers/media/platform/mtk-vcodec/venc_drv_base.h  |   62 +
> >  drivers/media/platform/mtk-vcodec/venc_drv_if.c    |  106 ++
> >  drivers/media/platform/mtk-vcodec/venc_drv_if.h    |  175 +++
> >  drivers/media/platform/mtk-vcodec/venc_ipi_msg.h   |  212 +++
> >  drivers/media/platform/mtk-vcodec/vp8_enc/Makefile |    6 +
> >  .../platform/mtk-vcodec/vp8_enc/venc_vp8_if.c      |  422 ++++++
> >  .../platform/mtk-vcodec/vp8_enc/venc_vp8_if.h      |  149 +++
> >  .../platform/mtk-vcodec/vp8_enc/venc_vp8_vpu.c     |  240 ++++
> >  .../platform/mtk-vcodec/vp8_enc/venc_vp8_vpu.h     |   28 +
> >  drivers/media/platform/mtk-vpu/Makefile            |    1 +
> >  drivers/media/platform/mtk-vpu/mtk_vpu.c           |  994 ++++++++++++++
> >  drivers/media/platform/mtk-vpu/mtk_vpu.h           |  167 +++
> >  include/uapi/linux/v4l2-controls.h                 |    4 +
> >  34 files changed, 6573 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/media/mediatek-vcodec.txt
> >  create mode 100644 Documentation/devicetree/bindings/media/mediatek-vpu.txt
> >  create mode 100644 drivers/media/platform/mtk-vcodec/Makefile
> >  create mode 100644 drivers/media/platform/mtk-vcodec/h264_enc/Makefile
> >  create mode 100644 drivers/media/platform/mtk-vcodec/h264_enc/venc_h264_if.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/h264_enc/venc_h264_if.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/h264_enc/venc_h264_vpu.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/h264_enc/venc_h264_vpu.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_drv.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_pm.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_intr.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_intr.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_pm.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_util.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/mtk_vcodec_util.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/venc_drv_base.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/venc_drv_if.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/venc_drv_if.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/venc_ipi_msg.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/vp8_enc/Makefile
> >  create mode 100644 drivers/media/platform/mtk-vcodec/vp8_enc/venc_vp8_if.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/vp8_enc/venc_vp8_if.h
> >  create mode 100644 drivers/media/platform/mtk-vcodec/vp8_enc/venc_vp8_vpu.c
> >  create mode 100644 drivers/media/platform/mtk-vcodec/vp8_enc/venc_vp8_vpu.h
> >  create mode 100644 drivers/media/platform/mtk-vpu/Makefile
> >  create mode 100644 drivers/media/platform/mtk-vpu/mtk_vpu.c
> >  create mode 100644 drivers/media/platform/mtk-vpu/mtk_vpu.h
> >  mode change 100644 => 100755 include/uapi/linux/v4l2-controls.h
> > 
>