Re: [PATCH v12 10/15] media: uapi: Add audio rate controls support

[Shengjiu Wang]

On Sat, Feb 17, 2024 at 5:57 PM Mauro Carvalho Chehab
<mchehab@xxxxxxxxxx> wrote:
>
> Em Thu, 18 Jan 2024 20:32:03 +0800
> Shengjiu Wang <shengjiu.wang@xxxxxxx> escreveu:
>
> > Add V4L2_CID_M2M_AUDIO_SOURCE_RATE and V4L2_CID_M2M_AUDIO_DEST_RATE
> > new IDs for rate control.
> >
> > Add V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET and
> > V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET for clock drift.
> >
> > Signed-off-by: Shengjiu Wang <shengjiu.wang@xxxxxxx>
> > ---
> >  .../media/v4l/ext-ctrls-audio-m2m.rst         | 20 +++++++++++++++++++
> >  drivers/media/v4l2-core/v4l2-ctrls-defs.c     |  6 ++++++
> >  include/uapi/linux/v4l2-controls.h            |  5 +++++
> >  3 files changed, 31 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> > index 82d2ecedbfee..de579ab8fb94 100644
> > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> > @@ -19,3 +19,23 @@ Audio M2M Control IDs
> >      The Audio M2M class descriptor. Calling
> >      :ref:`VIDIOC_QUERYCTRL` for this control will
> >      return a description of this control class.
> > +
> > +.. _v4l2-audio-asrc:
> > +
> > +``V4L2_CID_M2M_AUDIO_SOURCE_RATE (integer menu)``
> > +    Sets the audio source sample rate, unit is Hz
> > +
> > +``V4L2_CID_M2M_AUDIO_DEST_RATE (integer menu)``
> > +    Sets the audio destination sample rate, unit is Hz
> > +
> > +``V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET (fixed point)``
> > +    Sets the offset from the audio source sample rate, unit is Hz.
> > +    The offset compensates for any clock drift. The actual source audio
> > +    sample rate is the ideal source audio sample rate from
> > +    ``V4L2_CID_M2M_AUDIO_SOURCE_RATE`` plus this fixed point offset.
> > +
> > +``V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET (fixed point)``
> > +    Sets the offset from the audio destination sample rate, unit is Hz.
> > +    The offset compensates for any clock drift. The actual destination audio
> > +    sample rate is the ideal source audio sample rate from
> > +    ``V4L2_CID_M2M_AUDIO_DEST_RATE`` plus this fixed point offset.
>
> Hmm... first of all, controls on V4L2 API can either be get or set.
> So, starting the sentence with "Set" sounds an assumption that may
> be wrong.

Ok, will update the description.
>
> Also, I would explain a little bit more about the frequency offset values,
> as clock drift adjustment on PCM streams is something that can be done
> using different approaches.
>
> I'm assuming that what you wanted here is to use it to check if the
> video and audio clocks have some drift, and reducing or increasing
> the audio sample rate dynamically to ensure that such drift will
> stay constraint to a maximum allowed drift measured in mili or nano
> seconds. So, userspace would be expected to be monitoring such drift
> and increasing/decreasing the sample frequency as needed to maintain
> such constraint.
>
> Is that the way such uAPI is expected to work?

Yes. Userspace should monitor the drift, get the offset based on the
common sample rate (8k, 44.1k, 48k...) then send it to the driver.
The offset is a fixed point. it is base on the patch:
https://patchwork.kernel.org/project/linux-media/patch/cec82507-ced9-4e7d-802c-04a40f84a4b4@xxxxxxxxx/

Best regards
Shengjiu Wang
>
> > diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> > index 2a85ea3dc92f..91e1f5348c23 100644
> > --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> > +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> > @@ -1245,6 +1245,8 @@ const char *v4l2_ctrl_get_name(u32 id)
> >
> >       /* Audio M2M controls */
> >       case V4L2_CID_M2M_AUDIO_CLASS:  return "Audio M2M Controls";
> > +     case V4L2_CID_M2M_AUDIO_SOURCE_RATE:    return "Audio Source Sample Rate";
> > +     case V4L2_CID_M2M_AUDIO_DEST_RATE:      return "Audio Destination Sample Rate";
> >       default:
> >               return NULL;
> >       }
> > @@ -1606,6 +1608,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> >       case V4L2_CID_COLORIMETRY_HDR10_MASTERING_DISPLAY:
> >               *type = V4L2_CTRL_TYPE_HDR10_MASTERING_DISPLAY;
> >               break;
> > +     case V4L2_CID_M2M_AUDIO_SOURCE_RATE:
> > +     case V4L2_CID_M2M_AUDIO_DEST_RATE:
> > +             *type = V4L2_CTRL_TYPE_INTEGER_MENU;
> > +             break;
> >       default:
> >               *type = V4L2_CTRL_TYPE_INTEGER;
> >               break;
> > diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> > index a8b4b830c757..30129ccdc282 100644
> > --- a/include/uapi/linux/v4l2-controls.h
> > +++ b/include/uapi/linux/v4l2-controls.h
> > @@ -3495,6 +3495,11 @@ struct v4l2_ctrl_av1_film_grain {
> >  #define V4L2_CID_M2M_AUDIO_CLASS_BASE  (V4L2_CTRL_CLASS_M2M_AUDIO | 0x900)
> >  #define V4L2_CID_M2M_AUDIO_CLASS       (V4L2_CTRL_CLASS_M2M_AUDIO | 1)
> >
> > +#define V4L2_CID_M2M_AUDIO_SOURCE_RATE       (V4L2_CID_M2M_AUDIO_CLASS_BASE + 0)
> > +#define V4L2_CID_M2M_AUDIO_DEST_RATE (V4L2_CID_M2M_AUDIO_CLASS_BASE + 1)
> > +#define V4L2_CID_M2M_AUDIO_SOURCE_RATE_OFFSET        (V4L2_CID_M2M_AUDIO_CLASS_BASE + 2)
> > +#define V4L2_CID_M2M_AUDIO_DEST_RATE_OFFSET  (V4L2_CID_M2M_AUDIO_CLASS_BASE + 3)
> > +
> >  /* MPEG-compression definitions kept for backwards compatibility */
> >  #ifndef __KERNEL__
> >  #define V4L2_CTRL_CLASS_MPEG            V4L2_CTRL_CLASS_CODEC
>
>
>
> Thanks,
> Mauro