Re: [RFCv2] Media Token API Spec.

[Sakari Ailus]

Hi Shuah,

Many thanks for the RFC, for and being so patient with my terrible review
schedule. My comments below.

On Tue, Nov 04, 2014 at 04:08:50PM -0700, Shuah Khan wrote:
> Hi Mauro,
> 
> Here is the RFC as promised. I also included the Media controller
> as a an alternative and captured the discussion in the thread on
> that topic. Please review.
> 
> -- Shuah
> 
> -----------------------------------------------------------------
> RFC Media Token API Specification
> 
> Let's start with a diagram of a media device (without IR, eeprom
> and others):
> 
> http://linuxtv.org/downloads/presentations/typical_hybrid_hardware.png
> 
> The dot lines represent the parts of the graph that are switched by
> the tuner, DMA or input select.
> 
> Please notice that the DMA engines, together with the stuff needed to
> control A/V switches is at one single chip. Changing the registers there
> can affect the other streams, specially on most sophisticated devices
> like cx231xx, where it even has a power management IP block that
> validates if a device to be turned on/off won't exceed the maximum
> drain current of 500mA. That's basically why we need to do a temporary
> lock alsa, dvb, v4l and IR drivers when doing certain changes.
> 
> Also, please notice that I2C buses that can be as slow as 10kbps
> are used to control for several devices, like:
>         - the tuner
>         - the Digital TV (DTV) demod
>         - Analog and/or Video demod (sometimes embedded at the main
>           chip)
>         - DTV demux (sometimes embedded at the main chip)
>         - The remote controller (sometimes embedded at the main chip)
> 
> For some devices, after powered on, or when certain parameters change, a
> new firmware (and sometimes a hardware reset) is required. The firmware
> size can be about 64KB or even bigger.
> 
> Also, the A/V switch it is actually two independent switches (or one
> switch for video and one audio mux for audio) that needs to be changed
> together when the source changes.
> 
> There are two components that are shared there between analog and
> digital: the tuner (where the signal is captured) and the DMA engine
> used to stream analog and Digital TV (dvb).
> 
> PS.: the diagram is over-simplified, as the tuner is just one of the
> possible inputs for the analog part of the device. Other possible
> inputs are S-Video, composite, HDMI, etc.
> 
> Sometimes, the audio DMA is also shared, e. g. just one stream comes
> from the hardware. It is up to the driver to split audio and video and
> send them to the V4L2 and ALSA APIs. This is the case of tm6000 driver.
> 
> Those shared components can be used either at analog or digital mode,
> but not at the same time.
> 
> Also, programming the V4L2 analog and audio DMA and demods should be
> done via V4L2 API, as this API allows the selection of the proper
> audio/video input (almost all devices have multiple analog inputs).
> 
> Please notice that, if the tuner is on digital mode, the entire analog
> path is disabled, including ALSA output.
> 
> If the tuner is on analog mode, both ALSA and V4L2 can work at the
> same time. However, during the period where the tuner firmware is
> loaded, and during the DMA configuration and input selection time,
> neither ALSA or V4L2 can stream. Such configuration/firmware load
> is commanded via V4L2 API, as ALSA knows nothing about tuner or
> input selection.
> 
> At a higher level the problem description is:
> 
> There are 3 different device files that get created to control
> tuner and audio functions on a media device. 4 drivers (dvb,
> v4l2, alsa, and the main usb driver for the usb device), and
> 3 core APIs (dvb-core, v4l-core, audio) that control the tuner
> and audio hardware and provide user API to these 3 device files.
> 
> The above driver model is simplified, there's 4th component for
> some drivers: the mceusb driver, that handles remote controllers.
> The mceusb handles the Microsoft Media Center Remote Control
> protocol. It supports stand alone remote controller devices, but
> it also supports a few USB devices that use a separate interface
> for IR.
> 
> There are currently some issues on cx231xx and mceusb, as both drivers
> can be used at the same time, but, when cx231xx sends certain commands,
> the mceusb IR polls fail. This is out of the scope of the audio lock,
> but it also needs to be addressed some day.
> 
> Most media user applications, drivers and the core have no knowledge
> of each other. The only thing that is common across all these drivers
> is the parent device for the main usb device which is controlled by
> the usb driver.
> 
> Some media user applications like MythTV can handle all 3 APIs,
> however, MythTV doesn't know how to associate ALSA, V4L2 and DVB
> devnodes that belong to the same device. If MythTV finds, 3 V4L2
> nodes, 3 ALSA nodes, and 1 DVB node, it doesn't know which device
> is associated with the DVB node.
> 
> Almost all applications that are aware of V4L2 API are also aware of
> ALSA API and may associate audio and video, as there is a way to
> associate it using sysfs. However, several applications don't use it.
> 
> The premise for the main design idea in this series is creating
> a common construct at the parent device structure that is visible
> to all drivers to act as a master access control (lock). Let's call
> this media token object with two sub-tokens one for tuner and another
> for audio.
> 
> Each of the APIS evolved separately, hence have their own backwards
> compatibility to maintain. Starting with v4l2:
> 
> V4L2 case:
> Multiple v4l2 applications are allowed to open /dev/video0 in
> read/write mode with no restrictions as long as the tuner is in
> analog mode. V4L2-core handles conflicting requests between v4l2
> applications. V4L2-core doesn't have the knowledge that the tuner
> is in use by a dvb and/or audio is in use. Individual drivers
> may have this knowledge as, except for one case (bttv driver),
> they share some data.
> 
> As soon as a V4L2 application starts, digital stream glitches and
> audio glitches.
> 
> DVB case:
> Multiple dvb applications can open the dvb device in read only mode.
> As soon an application open the device read/write mode a separate
> kthread is kicked off to handle the request. Only one application
> can open the device in read/write mode. There's no issue with ALSA
> in R/O mode, as the application is not allowed to modify anything
> with the stream. This is used only to monitor an already opened
> device in R/W mode.
> 
> Similar to V4L2-core case, dvb-core doesn't have any knowledge that
> the tuner is in use by v4l2 and/or audio is in use. As soon as a
> dvb application starts v4l2 video glitches and audio glitches.
> 
> Audio case:
> Same scenario is applicable to audio application. When a v4l2 or dvb
> application starts, audio application gets impacted.
> 
> Problems to address:
> 
> Dvb owns tuner and audio: another dvb, v4l2 application and
>                           ALSA application should detect
>                           tuner/audio busy right away and exit.
>                           Dvb applications don't use audio node,
>                           however, devices can't use audio hardware
>                           while in DVB mode.
> 
> V4l2 owns tuner and audio: dvb should detect tuner/audio busy
>                            right away and exit.
>                            The V4L2-core should only hold the token for
>                            the required time to initialize the device
>                            and/or load the firmware.
>                            ALSA applications should wait for v4L2-core
> 			   to finish programming at audio, and should
>                            keep working after that.
> 
> Audio owns audio: dvb applications should detect audio busy and
> exit. V4L2 applications should work. However, when certain V4L2
> ioctls are issued, the audio device driver should not send any
> command to the hardware. After such commands, the audio mixers
> may change. This is why two tokens are necessary, one for tuner
> and another for audio.
> 
> Because of the above mentioned difference in behavior between dvb
> and v4l applications when audio is busy, two tokens (one for tuner
> and another for audio) are necessary and audio token lock should not
> be held at ALSA open/close.
> 
> Special cases:
> 
> Dvb applications access tuner in exclusive mode. i.e only one dvb
> application at a time is allowed to open the device read/write mode.
> Dvb applications don't use audio node, however, devices can't use
> audio hardware while in DVB mode. Dvb applications receive data as
> MPEG-TS, using a separate device node. The same DMA engine that
> provides video (and, sometimes audio) is used by the DVB device
> node, making it inaccessible to audio applications while tuner
> is in DVB mode. Hence, the need to prevent audio applications from
> accessing audio node when tuner is in DVB mode. As a result, dvb-core
> will have to hold tuner and audio tokens so v4l2-core and ALSA know
> that audio is not available. Dvb disables audio hardware so it could
> be powered-off in some cases.
> 
> Audio applications access audio in exclusive mode. i.e only one audio
> application at a time is allowed to open the device in read/write mode.
> Audio applications create threads and thread closes and re-opens the
> audio device. Threads can do this and hence something that higher level
> construct has to allow. Audio application has to hold audio token so
> dvb and v4l2 know that it is in use.
> 
> V4l2 applications access tuner and audio in shared v4l2 mode.
> i.e several v4l2 processes and threads could use tuner and audio
> at the same time. V4L2 core handles concurrency. There's just
> one file handler with full control to start/stop stream at V4L2
> side. The higher level construct should not break the ability of
> multiple v4l2 applications to access tuner and audio in shared
> mode, and disallow dvb and audio applications access when they
> are in use by the V4L2-core.
> 
> Dvb-core when it gets the tuner, it should also obtain audio right
> away. v4l2-core when it gets the tuner, it should get the audio at
> the same time. When dvb-core has the tuner, v4l2 shouldn't get it
> and vice versa.
> 
> When dvb-core has the audio locked, audio application should detect
> condition and stop streaming, as part of the hardware can be powered
> off. It can only return opening the device after DVB releases audio
> token.
> 
> When v4l2-core has audio locked, audio application should detect the
> condition and stop sending commands to audio hardware. It can only
> resume audio access after V4L2 releases audio token.
> 
> Open issues:
> During testing, snd_pcm_lib_ioctls are coming from dvb application.
> It is likely that these are related to the audio output and not audio
> capture or the application in question is an hybrid one. This issue
> needs further investigation.
> 
> Alternatives: (proposed by Sakari Ailus)
> Can Media controller be used to solve the problem?
> 
> The usage of the media controller for this specific usage is that
> we should not force userspace applications to be aware of the
> media controller just because of hardware locking.
> 
> Currently, media entities may only be entities bound to a given
> subsystem, but likely need to change media controller for complex
> embedded DVB device support ...
> 
> In case of the Media controller, mutual exclusion of different users
> is currently performed by adding the entities to a pipeline and
> incrementing the streaming count once streaming is enabled --- on
> different interfaces streaming may mean a different thing.
> 
> However, we'll still need to find a way for ALSA to prevent it to use
> the audio demod and DMA engine that will be powered off when DVB is
> streaming.
> 
> The Media controller interface does not handle serializing potential
> users that may wish to configure the device. Handling serializing is
> necessary if Media controller is extended instead of pursuing Media
> Token API to solve the problem.
> 
> Reconfiguring the DMA engine and some other registers via V4L2 API
> should be blocked. The same applies to firmware load, if the device
> is using tuner input for analog TV.
> 
> If we use the media controller, we'll need to add a state to it,
> to indicate that a block at the pipeline is being reconfigured.
> 
> It is dependent on Media Controller adoption on ALSA as well.

Thank you for the detailed description of the problem domain.

Using Media controller for this at this point isn't straightforward, and
especially for existing drivers for which the current APIs serve the
existing devices well enough, perhaps not the best solution even when the
missing pieces were implemented.

More complex devices, though, still need MC in order to control them in a
meaningful way. If a tuner is connected into one, the media token framework
wouldn't help there. In other words, we'll need something for such devices
as well. That'd be proper MC support for DVB and ALSA, but it'll be a
separate discussion.

One big upside in this patchset is that it does not change the user space
interface.

One concern I have is how generic this framework really is. Do you see
potential use cases outside the current one?

I'd move this under drivers/media, and possibly think of the naming of the
framework a little bit.

I'll review the rest of the v2.

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@xxxxxx	XMPP: sailus@xxxxxxxxxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/