Re: [RFC PATCH 0/3] METADATA design using V4l2 Request API
From: Hans Verkuil
Date: Tue May 26 2020 - 03:03:53 EST
On 26/05/2020 08:31, dikshita@xxxxxxxxxxxxxx wrote:
> A gentle reminder for the review.
It's on my TODO list. I hope to get to it this week or next week at the
> On 2020-05-08 11:51, Dikshita Agarwal wrote:
>> There are many commercialized video use cases which needs metadata info
>> to be circulated between v4l2 client and v4l2 driver.
>> METADATA has following requirements associated:
>> âMetadata is an optional info available for a buffer. It is not
>> mandatorily for every buffer.
>> For ex. consider metadata ROI (Region Of Interest). ROI is specified
>> by clients to indicate
>> the region where enhanced quality is desired. This metadata is given
>> as an input information
>> to encoder output plane. Client may or may not specify the ROI for a
>> frame during encode as
>> an input metadata. Also if the client has not provided ROI metadata
>> for a given frame,
>> it would be incorrect to take the metadata from previous frame. If the
>> data and
>> metadata is asynchronous, it would be difficult for hardware to decide
>> if it
>> needs to wait for metadata buffer or not before processing the input
>> frame for encoding.
>> âSynchronize the buffer requirement across both the video node/session
>> (incase metadata is being processed as a separate v4l2 video
>> This is to avoid buffer starvation.
>> âAssociate the metadata buffer with the data buffer without adding any
>> pipeline delay
>> in waiting for each other. This is applicable both at the hardware
>> side (the processing end)
>> and client side (the receiving end).
>> âLow latency usecases like WFD/split rendering/game streaming/IMS have
>> sub-50ms e2e latency
>> requirements, and it is not practical to stall the pipeline due to
>> inherent framework latencies.
>> High performance usecase like high-frame rate playback/record can
>> lead to frame loss during any pipeline latency.
>> To address all above requirements, we used v4l2 Request API as
>> As an experiment, We have introduced new control
>> to contain the METADATA info. Exact controls can be finalized once the
>> interface is discussed.
>> For setting metadata from userspace to kernel, let say on encode output
>> following code sequence was followed
>> 1. Video driver is registering for media device and creating a media
>> node in /dev
>> 2. Request fd is allocated by calling MEDIA_IOC_REQUEST_ALLOC IOCTL on
>> media fd.
>> 3. METADATA configuration is being applied on request fd using
>> VIDIOC_S_EXT_CTRLS IOCTL
>> and the same request fd is added to buf structure structure before
>> calling VIDIOC_QBUF on video fd.
>> 4. The same request is queued through MEDIA_REQUEST_IOC_QUEUE IOCTL to
>> driver then, as a result
>> to which METADATA control will be applied to buffer through S_CTRL.
>> 5. Once control is applied and request is completed,
>> MEDIA_REQUEST_IOC_REINIT IOCTL is called
>> to re-initialize the request.
>> We could achieve the same on capture plane as well by removing few
>> checks present currently
>> in v4l2 core which restrict the implementation to only output plane.
>> We profiled below data with this implementation :
>> 1. Total time taken ( round trip ) for setting up control data on video
>> with VIDIOC_S_EXT_CTRLS, QBUF and Queue Request: 737us
>> 2. Time taken for first QBUF on Output plane to reach driver with
>> REQUEST API enabled (One way): 723us
>> 3. Time taken for first QBUF on Output plane to reach driver without
>> REQUEST API (One way) : 250us
>> 4. Time taken by each IOCTL to complete ( round trip ) with REQUEST
>> API enabled :
>> a. VIDIOC_S_EXT_CTRLS : 201us
>> b. VIDIOC_QBUF : 92us
>> c. MEDIA_REQUEST_IOC_QUEUE: 386us
>> Kindly share your feedback/comments on the design/call sequence.
>> Also as we experimented and enabled the metadata on capture plane as
>> well, please comment if any issue in
>> allowing the metadata exchange on capture plane as well.
>> Reference for client side implementation can be found at .
>> Dikshita Agarwal (3):
>> Register for media device
>> - Initialize and register for media device
>> - define venus_m2m_media_ops
>> - Implement APIs to register/unregister media controller.
>> Enable Request API for output buffers
>> - Add dependency on MEDIA_CONTROLLER_REQUEST_API in Kconfig.
>> - Initialize vb2 ops buf_out_validate and buf_request_complete.
>> - Add support for custom Metadata control
>> - Implemeted/Integrated APIs for Request setup/complete.
>> Enable Request API for Capture Buffers
>> drivers/media/common/videobuf2/videobuf2-v4l2.c | 4 +-
>> drivers/media/platform/Kconfig | 2 +-
>> drivers/media/platform/qcom/venus/core.h | 36 ++++
>> drivers/media/platform/qcom/venus/helpers.c | 247
>> drivers/media/platform/qcom/venus/helpers.h | 15 ++
>> drivers/media/platform/qcom/venus/venc.c | 63 +++++-
>> drivers/media/platform/qcom/venus/venc_ctrls.c | 61 +++++-
>> drivers/media/v4l2-core/v4l2-ctrls.c | 10 +
>> drivers/media/v4l2-core/v4l2-mem2mem.c | 17 +-
>> include/media/v4l2-ctrls.h | 1 +
>> include/media/venus-ctrls.h | 22 +++
>> 11 files changed, 465 insertions(+), 13 deletions(-)
>> create mode 100644 include/media/venus-ctrls.h