Re: [RFCv1] Media Token API needs - Was: Re: [alsa-devel] [PATCH v2 5/6] sound/usb: pcm changes to use media token api

[Shuah Khan]

On 10/28/2014 05:42 PM, Mauro Carvalho Chehab wrote:
> Hi Shuah,
> 
> I'm understanding that you're collecting comments to write a RFC with the
> needs by the media token, right?
> 
> I'm sending you my contributions to such text. See enclosed.
> 
> I suggest to change the subject and submit this on a separate thread, after
> we finish the review of such document. Anyway, I'm changing the subject
> of this Thread to reflect that.
> 
> Regards,
> Mauro
> 
> Em Tue, 28 Oct 2014 15:15:43 -0600
> Shuah Khan <shuahkh@xxxxxxxxxxxxxxx> escreveu:
> 
>> On 10/27/2014 06:52 AM, Mauro Carvalho Chehab wrote:
>>> Em Sun, 26 Oct 2014 09:27:40 +0100
>>> Takashi Iwai <tiwai@xxxxxxx> escreveu:
>>>
>>
>>>
>>> Hmm... this is actually more complex than that. V4L2 driver doesn't
>>> know if ALSA is streaming or not, or even if ALSA device node is opened
>>> while he is touching at the hardware configuration or changing the
>>> state. I mean: it is not an error to set the hardware. The error only
>>> happens if ALSA and V4L2 tries to do it at the same time on an incompatible
>>> way.
>>>
>>> Also, this won't work for DVB, as on DVB this is really an exclusive
>>> lock that would prevent both ALSA and V4L2 drivers to stream while in
>>> DVB mode.
>>>
>>> Implementing it with a lock seems to be the best approach, at least on
>>> my eyes.
>>>
>>>> That said, we should go back and start discussing the design goal at
>>>> first.
>>>
>>> Surely.
>>
>> This is long, however, hoping it will describe the problem and
>> solution that is being pursued in detail:
> 
> Before starting with the description, this is the simplified diagram of
> a media device (without IR, eeprom and other beasts):
> 
>   +-----------------------------------------------------------------------------------------+
>   |                                                                                         |
>   |                   +----------------+     +------------------+-------+----------------+  |
>   |                   |  demod_video   | <-- |      analog      | tuner |     digital    |  |
>   |                   +----------------+     +------------------+-------+----------------+  |
>   |                     |                      |                           |                |
>   |                     |                      |                           |                |
>   v                     v                      v                           v                |
> +--------------+-----+-----------------+     +------------------+        +---------------+  |
> |     dvb      | DMA |      analog     |     |   demod_audio    |        | digital_demux | -+
> +--------------+-----+-----------------+     +------------------+        +---------------+
>   |                     |                      |
>   |                     |                      |
>   v                     v                      v
> +--------------+      +----------------+     +------------------+
> | devnode dvr0 |      | devnode video0 |     |    audio DMA     |
> +--------------+      +----------------+     +------------------+
>                                                |
>                                                |
>                                                v
>                                              +------------------+
>                                              | devnode pcmC1D0c |
>                                              +------------------+
> 
> There are two components that are shared there between analog and digital:
> the tuner (where the signal is captured) and the DMA engine used to stream
> analog and Digital TV (dvb).
> 
> PS.: the diagram is over-simplified, as the tuner is just one of the possible
> inputs for the analog part of the device. Other possible inputs are S-Video,
> composite, HDMI, etc.
> 
> Sometimes, the audio DMA is also shared, e. g. just one stream comes from
> the hardware. It is up to the driver to split audio and video and send
> them to the V4L2 and ALSA APIs. This is the case of tm6000 driver.
> 
> Those shared components can be used either at analog or digital mode,
> but not at the same time.
> 
> Also, programming the V4L2 analog and audio DMA and demods should be done
> via V4L2 API, as this API allows the selection of the proper audio/video
> input (almost all devices have multiple analog inputs).
> 
> Please notice that, if the tuner is on digital mode, the entire analog
> path is disabled, including ALSA output.
> 
> If the tuner is on analog mode, both ALSA and V4L2 can work at the
> same time. However, during the period where the tuner firmware is
> loaded, and during the DMA configuration and input selection time,
> neither ALSA or V4L2 can stream. Such configuration/firmware load
> is commanded via V4L2 API, as ALSA knows nothing about tuner or
> input selection.
> 
>>
>> At a higher level the problem description is:
>>
>> There are 3 different device files that get created to control
>> tuner and audio functions on a media device. 3 drivers (dvb,
>> v4l2, alsa), and 3 core apis (dvb-core, v4l-core, audio) that
>> control the tuner and audio hardware and provide user api to
>> these 3 device files.
> 
> 
> There's actually a 4th component for some drivers: the mceusb driver,
> that handles remote controllers. The mceusb handles the Microsoft 
> Media Center Remote Control protocol. It supports standalone remote
> controller devices, but it also supports a few USB devices that use
> a separate interface for IR.
> 
> There are currently some issues on cx231xx and mceusb, as both drivers 
> can be used at the same time, but, when cx231xx sends certain commands, 
> the mceusb IR polls fail. This is out of the scope of the audio lock,
> but it also needs to be addressed some day.
> 
>> User applications, drivers and the core have no knowledge of each
>> other. The only thing that is common across all these drivers is
>> the parent device for the main usb device which is controlled by
>> the main usb driver.
> 
> I would add that there are user applications that can handle all
> 3 APIs like MythTV. But, at least MythTV doesn't know how to associate
> ALSA, V4L2 and DVB devnodes that belong to the same device.
> 
> I mean: if MythTV finds, let's say, 3 V4L2 nodes, 3 ALSA nodes, 
> and 1 DVB node, it doesn't know what device is associated with the
> DVB node.
> 
> Almost all applications that are aware of V4L2 API are also aware of
> ALSA API and may associate audio and video, as there is a way to 
> associate it using sysfs. However, several apps don't use it.
> 
>> The premise for the main design idea in this series is creating
>> a common construct at the parent device structure that is visible
>> to all drivers to act as a master access control (lock). Let's call
>> this media token object with two sub-tokens one for tuner and another
>> for audio.
>>
>> Each of the apis evolved separately, hence have their own backwards
>> compatibility to maintain. Starting with v4l2:
>>
>> v4l2 case:
>> Multiple v4l2 applications are allowed to open /dev/video0 in
>> read/write mode with no restrictions as long as the tuner is in
>> analog mode. v4l2 core handles conflicting requests between v4l2
>> applications. 
> 
>> It doesn't have the knowledge that the tuner is in
> 
> To be clear: "It" here refers to v4l2 core. The drivers may have this
> knowledge as, except for one case (bttv driver), they share some data.
> 
>> use by a dvb and/or audio is in use. As soon as a v4l2 application
>> starts, digital stream glitches and audio glitches.
>>
>> dvb case:
>> Multiple dvb applications can open the dvb device in read only mode.
> 
> There's no issue with ALSA on R/O mode, as the application is not
> allowed to modify anything at the stream. This is used only to monitor
> an already opened device in R/W mode.
> 
>> As soon an application open the device read/write mode a separate
>> kthread is kicked off to handle the request. Only one application
>> can open the device in read/write mode. 
> 
>> Similar to v4l2 case,
> 
> s/v4l2/v4l2 core/
> 
>> dvb-core doesn't have any knowledge that the tuner is in use by
>> v4l2 and/or audio is in use. As soon as a dvb application starts v4l2
>> video glitches and audio glitches.
>>
>> audio case:
>> Same scenario is applicable to audio application. When a v4l2 or dvb
>> application starts, audio application gets impacted.
>>
>> Problems to address:
>>
>> dvb owns tuner and audio: another dvb, v4l2 app and audio app should
>>                           detect tuner/audio busy right away and exit.
>>
>> v4l2 owns tuner and audio: another dvb and audio app should detect
>>                            tuner/audio busy right away and exit.
> 
> Actually, no: audio should not exit. The V4L2 should only hold the
> token for the required time to initialize the device and/or load the
> firmware. ALSA applications should wait for V4L2 to finish
> programming at audio, and should keep working after that.
> 
>>                            v4l2 app can continue to use it until it
>>                            tries to change the tuner/audio state.
>>
>>
>> audio owns audio: dvb and v4l2 apps should detect audio busy and exit.
> 
> Actually no. It is, instead:
> 
> audio owns audio: dvb apps should detect audio busy and exit.
> V4L2 apps should work. However, when certain V4L2 ioctls are issued, 
> the audio device driver should not send any command to the hardware.
> After such commands, the audio mixers may change.
> 
> We need two separate tokens because of that: the behavior is different.
> 
> This is basically why we need two separate tokens, and because we cannot
> implement locking at ALSA open/close.
> 
>>
>> Special cases:
>>
>> dvb apps. access tuner and audio in exclusive mode. i.e only one dvb app.
>> at a time is allowed to open the device read/write mode.
> 
> To be clearer: dvb apps won't use the audio node, but audio should be blocked,
> as the devices can't use audio while in DVB mode.
> 
>> As dvb apps.
>> create threads to handle audio and video, 
> 
> No. DVB apps don't handle audio/video. It receives data as MPEG-TS,
> using a separate device node. Yet, the same DMA engine that provides
> video (and, sometimes audio) is used by the DVB devnode.
> 
>> all threads in that group
>> should be allowed by the higher level construct to access the tuner and
>> audio. dvb application will have to hold tuner and audio tokens so v4l2
>> and audio apps. know they are in use.
>>
>> audio apps. access audio in exclusive mode. i.e only one audio app. at
>> a time is allowed to open the device in read/write mode. Audio apps.
>> create threads and thread closes and re-opens the audio device. Threads
>> can do this and hence something that higher level construct has to allow.
>> audio app. has to hold audio token so dvb and v4l2 know that it is in use.
>> (Note: I am not sure if I have the audio scenario right)
>>
>> v4l2 apps. access tuner and audio in shared v4l2 mode. i.e several v4l2
>> processes and threads could use tuner and audio at the same time. The
>> higher level construct has to allow multiple v4l2 apps. to access and
>> disallow dvb and audio apps. access when they are in use by v4l2.
> 
> Actually, V4L2 core handles concurrency. There's just one file handler
> with full control to start/stop stream at V4L2 side.
> 
>>
>> Adding to this, both dvb and v4l2 open audio device and make snd pcm
>> capture callbacks.
> 
> Huh? DVB won't need to touch at PCM capture callbacks. It should just
> avoid audio PCM capture to stream while in DVB mode.
> 
>> There is no way to tell if dvb or v4l2 or audio
>> app is the one that is making this request.
> 
>> dvb app would like audio
>> in exclusive mode allowing only one process and its threads to access
>> it.
> 
> No. It just wants to disable the part of the hardware that can now
> be powered off.

Hmm. I am seeing some snd_pcm_lib_ioctls coming from dvb application.
>From what you are saying, these could be for poweroff.

> 
>> v4l2 on the other hand would like audio in shared state accessible
>> to all v4l2 processes. 
> 
>> If dvb-core and v4l2-core get tuner and audio
>> tokens at the same time, the window for having tuner token and not
>> getting audio token go down.
> 
> No. It should not be allowed that both dvb-core and v4l2-core to get
> the tokens at the same time. This is an exclusive lock.

I should have explained this better. What I meant was that dvb-core when
it gets the tuner, it should also obtain audio right away. v4l2-core
when it gets the tuner, it should get the audio at the same time. When
dvb-core has the tuner, v4l2 shouldn't get it and vice versa.

> 
>> In dvb case when dvb device is opened in read/write mode, and v4l2
>> case when an app. tries to change the status. Audio callbacks have to
>> detect if audio is busy, if not which mode to request the token in.
> 
> Huh?

ok again bad explaining on my part. Let me try again. When dvb-core has
the audio locked, audio application should detect the condition and take
appropriate action. When v4l2-core has audio locked, audio application
should detect the condition and take appropriate action.

> 
>> For dvb and audio app. cases, the audio token should be requested in
>> exclusive mode and in v4l2 case shared mode. The logic for requesting
>> audio token will have to be try to get in exclusive mode, if fails,
>> try to get in shared mode, and if that fails give up.
> 
> Huh?

Consider this flow:

step 1: dvb-core locks tuner and audio.
step 2: audio ioctl is initiated (from the application)
step 3: If dvb-core has the audio locked, how does alsa know if it can
        proceed with the ioctl request or not?

> 
>>
>> Current status:
>> Combining patch v1 and patch v2 designs by allowing shared mode token
>> hold for v4l2, and deciding on where to hold audio token from
>> alsa driver will solve the above conflict scenarios. That said, the
>> question is "is this the right approach?" or are there other ways to
>> solve the problem. One thing is clear, we need some common higher level
>> construct for all the device drivers and dvb, v4l2, and audio ioctls,
>> callbacks etc, to detect the hardware is in use.
> 
> I think that the current status is that we need to finish the spec
> first. Then check if the patches are doing what's above.
> 
> It seems that we agree to not agree at the requirements so far ;)

I would say it is more of not having the same understanding of
the requirements as opposed to wanting to agree to disagree.

Right I agree that developing a clear RFC spec will help us all be
on the same page while I continue to find solution for this problem,
and continuing work towards patch v3.

I will compile what we discussed so far in RFC and send it out for
review in a day or two. I will include your diagrams and the scenarios
I put together with your corrections in it.

-- Shuah


-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@xxxxxxxxxxxxxxx | (970) 217-8978
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/