Re: [PATCH v2 7/8] media: hevc: Add scaling matrix control

[Ezequiel Garcia]

On Mon, 2021-06-14 at 09:43 +0200, Benjamin Gaignard wrote:
> 
> Le 14/06/2021 à 09:27, Hans Verkuil a écrit :
> > On 10/06/2021 17:44, Benjamin Gaignard wrote:
> > > HEVC scaling lists are used for the scaling process for transform
> > > coefficients.
> > > V4L2_HEVC_SPS_FLAG_SCALING_LIST_ENABLED has to set when they are
> > > encoded in the bitstream.
> > Comparing H264 with HEVC I noticed that the corresponding flag for H264 is
> > called V4L2_H264_PPS_FLAG_SCALING_MATRIX_PRESENT.
> > 
> > Should those names be aligned? Also, it is part of PPS for H264 and SPS in HEVC,
> > is that difference correct?
> 
> In ITU specifications ("7.4.3.2.1 General sequence parameter set RBSP semantics") this flag is define like that:
> scaling_list_enabled_flag equal to 1 specifies that a scaling list is used for the scaling process for transform coefficients.
> scaling_list_enabled_flag equal to 0 specifies that scaling list is not used for the scaling process for transform coefficients.
> 
> So for me the naming is correct.
> 

The bitstream is really parsed in userspace (gstreamer, ffmpeg, chromium).
Not all bitstream parameters need to be passed as-is, because the kernel
is actually a sort of low-level layer in the decoder stack.

I think it's probably most appropriate to follow our H.264 interface
semantics, see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=54889c51b833d236228f983be16212fbe806bb89.

The H.264 story goes like this:

* Default scaling list is used for decoding, but can be modified
  by a bitstream-provided scaling list.
   
* The scaling list modification can be in SPS or in PPS.

* For each frame, the SPS and PPS headers will tell if
  a modified scaling list must be used for decoding,
  and if it's provided in the SPS header or the PPS header.

The userspace parser must take care of this, and pass
a scaling matrix to the kernel via V4L2_CID_MPEG_VIDEO_H264_SCALING_MATRIX,
setting PPS_FLAG_SCALING_MATRIX_PRESENT.

This flag is at the PPS control, so the scaling modification
can be applied on each frame.

In other words, the kernel interface is simpler, it just
receives a scaling matrix and a flag telling if it must be used or not. 

We should probably clarify the documentation, so this process is more clear.

>From the HEVC spec, it seems the process should be similar.
The only difference is the SPS "scaling_list_enabled_flag" parameter,
which doesn't exist in H.264.

Nicolas: what do you think?

Thanks,
Ezequiel