[RFC v4 00/17] V4L2 Explicit Synchronization support

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@xxxxxxxxxxxxx>

Hi,

I renamed this back to RFC as many things are still under
discussion/open but it integrates all comments received on the
previous round[1].

My main goal now is to fit as many use cases as possible into this
Explicit Synchronization implementation, but first I'd like to recap
some detail of the implementation, as some pieces changed since the last
patchset, that means the explanation on previous cover letters aren't
entirely valid anymore.

Explicit Synchronization allows us to control the synchronization of
shared buffers from userspace by passing fences to the kernel and/or
receiving them from it. Fences passed to the kernel are named in-fences
and the kernel should wait them to signal before using the buffer, i.e.,
queueing it to the driver. On the other side, the kernel can create
out-fences for the buffers it queues to the drivers, out-fences signal
when the driver is finished with buffer, that is, the buffer is ready.

Before we talk about more details, lets differentiate ordering concepts,
there is two orthogonal places in the pipeline where buffers may or may
not be ordered. Inside the drivers and inside vb2 core:

* Ordering in the driver: In this RFC to use out-fences v4l2 drivers
  need to guarantee ordering of the buffers inside the driver, that is,
  buffers should become done (think calling vb2_buffer_done() here) in
  the same order they are queued to the driver. Drivers should set
  q->ordered_in_driver to 1 if they can keep ordering. (See patch 08/17)

* Ordering in vb2: Some queues (OUTPUT queues and some m2m devices)
  requires that the order we queue buffers to the driver do not change
  in relation to the order they were queued from userspace. So that is
  marked in the code by q->is_output or q->ordered_in_vb2 and this
  implementation can work with both ordered and not-ordered queues at the
  vb2 level. (See patch 09/17)

The in-fences are communicated at the VIDIOC_QBUF ioctl using the
V4L2_BUF_FLAG_IN_FENCE buffer flag and the fence_fd field. If an
in-fence needs to be passed to the kernel, fence_fd should be set to the
fence file descriptor number and the V4L2_BUF_FLAG_IN_FENCE should be
set as well. If ordered in vb2 needs to be kept we use fence arrays (see
patch 12 and 13)

Out-fence for a given buffer is enabled if the V4L2_BUF_FLAG_OUT_FENCE
flag is passed in the QBUF() call and then communicated to userspace via
the newly created V4L2_EVENT_OUT_FENCE event. The event is queued for
userspace pick up when the buffer is queued to the driver, or if the
queue is ordered both in the driver and vb2 we send the event right
away. (see patch 14 to 16)

Use cases
---------

I believe use cases like camera app preview, photoshoot, video record,
m2m color converters and scalers (with ordered queues) and output
devices are covered by this RFC, but others like synchronized
Audio/Video transmission doesn't seem to benefit too much from fences.
Talking to Nicolas he expressed that fences are not really useful for
such cases because we still need to sync with the Audio side, so may
be we need wait at least for a start of streaming signal or something
like that.  For m2m encoders, we would be in a similar situation
needing to know bytesused beforehand. I want to hear what others think
here.

WIP on V4L2 event
-----------------

As the V4L2 event part is not the crucial part here, the current
changes there to allow install the fence_fd at DQ_EVENT() are to
be considered PoC. Suggestions are welcome. (See patch 4)

Main Changes on v4
-------------------

* Keep ordering in vb2 of buffers queued from userspace with
  in-fence attached. This is necessary for OUTPUT and some m2m
  queues.

* V4L2_EVENT_BUF_QUEUED event was renamed to
  V4L2_EVENT_OUT_FENCE and is only sent when an out-fence
  exists.

* For queues that are both ordered in vb2 and in driver the
  OUT_FENCE event is sent out immediately without waiting for
  the buffer to be queued on the driver

* The out-fence FD is only installed at DQ_EVENT(), but this
  implementation needs improvement. See 'Open Questions'

* Refactor slightly event subscription for v4l2 driver, now
  there is a v4l2_event_subscribe_v4l2() that will subscribe for
  event that all devices need.

* Protect dma_fence_remove_callback() with the fence lock.

Only the most important changes are detailed here, for a full
list of them, see each individual patch.

Open Questions
--------------

* synchronized Audio/Video transmission and m2m encoders use
  cases discussed above

* V4L2 event: Improve solution to install the fd and figure how
  to clean up a event if userspace don't pick it up.

Test tool
---------

Test tool I've been using can be found at:
https://git.collabora.com/cgit/user/padovan/v4l2-test.git/

Regards,

Gustavo

--
[1] https://lkml.org/lkml/2017/9/7/489

Gustavo Padovan (16):
  [media] v4l: create v4l2_event_subscribe_v4l2()
  [media] v4l: use v4l2_subscribe_event_v4l2() on vtables
  [media] v4l: use v4l2_subscribe_event_v4l2() on drivers
  WIP: [media] v4l2: add v4l2_event_queue_fh_with_cb()
  [media] v4l: add V4L2_EVENT_OUT_FENCE event
  [media] vb2: add .send_out_fence() to notify userspace of out_fence_fd
  [media] vivid: assign the specific device to the vb2_queue->dev
  [media] vb2: add 'ordered_in_driver' property to queues
  [media] vb2: add 'ordered_in_vb2' property to queues
  [media] vivid: mark vivid queues as ordered_in_driver
  [media] vb2: check earlier if stream can be started
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add infrastructure to support out-fences
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behaviour

Javier Martinez Canillas (1):
  [media] vb2: add videobuf2 dma-buf fence helpers

 Documentation/media/uapi/v4l/buffer.rst            |  15 ++
 Documentation/media/uapi/v4l/vidioc-dqevent.rst    |  23 ++
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  31 +++
 Documentation/media/videodev2.h.rst.exceptions     |   1 +
 drivers/media/common/saa7146/saa7146_video.c       |   4 +-
 drivers/media/dvb-frontends/rtl2832_sdr.c          |   2 +-
 drivers/media/pci/bt8xx/bttv-driver.c              |   4 +-
 drivers/media/pci/cobalt/cobalt-v4l2.c             |   2 +-
 drivers/media/pci/cx18/cx18-ioctl.c                |   2 +-
 drivers/media/pci/cx23885/cx23885-video.c          |   2 +-
 drivers/media/pci/cx25821/cx25821-video.c          |   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c            |   2 +-
 drivers/media/pci/cx88/cx88-video.c                |   4 +-
 drivers/media/pci/ivtv/ivtv-ioctl.c                |   2 +-
 drivers/media/pci/meye/meye.c                      |   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c        |   2 +-
 drivers/media/pci/saa7134/saa7134-video.c          |   4 +-
 drivers/media/pci/saa7164/saa7164-encoder.c        |   2 +-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c         |   2 +-
 drivers/media/pci/sta2x11/sta2x11_vip.c            |   2 +-
 drivers/media/pci/tw5864/tw5864-video.c            |   2 +-
 drivers/media/pci/tw68/tw68-video.c                |   2 +-
 drivers/media/pci/tw686x/tw686x-video.c            |   2 +-
 drivers/media/pci/zoran/zoran_driver.c             |   2 +-
 drivers/media/platform/am437x/am437x-vpfe.c        |   2 +-
 drivers/media/platform/atmel/atmel-isc.c           |   2 +-
 drivers/media/platform/atmel/atmel-isi.c           |   2 +-
 drivers/media/platform/coda/coda-common.c          |   2 +-
 drivers/media/platform/fsl-viu.c                   |   2 +-
 drivers/media/platform/marvell-ccic/mcam-core.c    |   2 +-
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       |   2 +-
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c |   2 +-
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c |   2 +-
 drivers/media/platform/pxa_camera.c                |   2 +-
 drivers/media/platform/qcom/venus/vdec.c           |   4 +-
 drivers/media/platform/qcom/venus/venc.c           |   2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c        |   2 +-
 drivers/media/platform/rcar_drif.c                 |   2 +-
 drivers/media/platform/rcar_fdp1.c                 |   2 +-
 drivers/media/platform/rcar_jpu.c                  |   2 +-
 drivers/media/platform/s3c-camif/camif-capture.c   |   2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c      |   2 +-
 drivers/media/platform/sti/hva/hva-v4l2.c          |   2 +-
 drivers/media/platform/stm32/stm32-dcmi.c          |   2 +-
 drivers/media/platform/ti-vpe/cal.c                |   2 +-
 drivers/media/platform/ti-vpe/vpe.c                |   2 +-
 drivers/media/platform/vim2m.c                     |   2 +-
 drivers/media/platform/vivid/vivid-core.c          |  15 +-
 drivers/media/platform/vivid/vivid-vid-out.c       |   2 +-
 drivers/media/radio/dsbr100.c                      |   2 +-
 drivers/media/radio/radio-cadet.c                  |   2 +-
 drivers/media/radio/radio-isa.c                    |   2 +-
 drivers/media/radio/radio-keene.c                  |   2 +-
 drivers/media/radio/radio-ma901.c                  |   2 +-
 drivers/media/radio/radio-miropcm20.c              |   2 +-
 drivers/media/radio/radio-mr800.c                  |   2 +-
 drivers/media/radio/radio-sf16fmi.c                |   2 +-
 drivers/media/radio/radio-si476x.c                 |   2 +-
 drivers/media/radio/radio-tea5764.c                |   2 +-
 drivers/media/radio/radio-tea5777.c                |   2 +-
 drivers/media/radio/radio-timb.c                   |   2 +-
 drivers/media/radio/si470x/radio-si470x-common.c   |   2 +-
 drivers/media/radio/si4713/radio-platform-si4713.c |   2 +-
 drivers/media/radio/si4713/radio-usb-si4713.c      |   2 +-
 drivers/media/radio/tea575x.c                      |   2 +-
 drivers/media/usb/airspy/airspy.c                  |   2 +-
 drivers/media/usb/au0828/au0828-video.c            |   2 +-
 drivers/media/usb/cpia2/cpia2_v4l.c                |   4 +-
 drivers/media/usb/cx231xx/cx231xx-417.c            |   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c          |   4 +-
 drivers/media/usb/em28xx/em28xx-video.c            |   4 +-
 drivers/media/usb/go7007/go7007-v4l2.c             |   2 +-
 drivers/media/usb/gspca/gspca.c                    |   2 +-
 drivers/media/usb/hackrf/hackrf.c                  |   2 +-
 drivers/media/usb/hdpvr/hdpvr-video.c              |   2 +-
 drivers/media/usb/msi2500/msi2500.c                |   2 +-
 drivers/media/usb/pwc/pwc-v4l.c                    |   2 +-
 drivers/media/usb/s2255/s2255drv.c                 |   2 +-
 drivers/media/usb/stk1160/stk1160-v4l.c            |   2 +-
 drivers/media/usb/stkwebcam/stk-webcam.c           |   2 +-
 drivers/media/usb/tm6000/tm6000-video.c            |   4 +-
 drivers/media/usb/usbvision/usbvision-video.c      |   4 +-
 drivers/media/usb/uvc/uvc_v4l2.c                   |   2 +-
 drivers/media/usb/zr364xx/zr364xx.c                |   2 +-
 drivers/media/v4l2-core/Kconfig                    |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c      |   4 +-
 drivers/media/v4l2-core/v4l2-event.c               |  45 +++-
 drivers/media/v4l2-core/videobuf2-core.c           | 239 +++++++++++++++++++--
 drivers/media/v4l2-core/videobuf2-v4l2.c           |  84 +++++++-
 .../vc04_services/bcm2835-camera/bcm2835-camera.c  |   2 +-
 include/media/v4l2-event.h                         |  15 ++
 include/media/videobuf2-core.h                     |  49 ++++-
 include/media/videobuf2-fence.h                    |  48 +++++
 include/uapi/linux/videodev2.h                     |  16 +-
 samples/v4l/v4l2-pci-skeleton.c                    |   2 +-
 95 files changed, 643 insertions(+), 125 deletions(-)
 create mode 100644 include/media/videobuf2-fence.h

-- 
2.13.6