Re: [PATCH v2] media: Documentation: Fix frame interval calculation for raw camera sensors

[Sakari Ailus]

Hi Jacopo,

On Fri, Mar 06, 2026 at 02:02:58PM +0100, Jacopo Mondi wrote:
> Hi Jai,
> 
> On Wed, Feb 25, 2026 at 02:19:35PM +0530, Jai Luthra wrote:
> > The previous frame interval formula used analogue crop dimensions. This
> > diverges from what most non-CCS camera sensor drivers do and what most
> > applications expect.
> >
> > The CCS specification describes under section "8.2.6 Line Length and
> > Frame Length" how the horizontal and vertical readout minimums can be
> > different when binning.
> >
> > We see the same thing in non-CCS sensor drivers, for example in imx219
> > the minimum FLL (frame length in lines) can be lower than the analogue
> > crop height when doing 2x2 or 4x4 binning. Similarly in imx283 HMAX
> > (line length) can be lower compared to the analogue crop width when
> > doing 2x2 or 3x3 binning.
> >
> > While the CCS driver defines the HBLANK/VBLANK controls with respect to
> > the analogue crop rectangle, with possibly negative values for blanking,
> > most other raw sensor drivers instead define them with respect to the
> > output dimensions (format defined on the source pad) sent on the bus.
> 
> Sakari: are negative blankins the way CCS is intended to be operated
> when LLP/FLL are smaller than the analog crop ?

Yes. The formula is rather simple.

> 
> >
> > So fix the frame interval calculation described in the documentation to
> > match the reality of existing drivers, separating out the CCS and
> > non-CCS cases. In future, with the common raw sensor model, this may
> > need further updates.
> >
> > While at it, also clarify the order of operations in the sensor pipeline
> > and give the "hardware" definition for frame interval using LLP (line
> > length in pixels) and FLL (frame length in lines) before the messy
> > definitions that use blanking controls.
> >
> > Signed-off-by: Jai Luthra <jai.luthra@xxxxxxxxxxxxxxxx>
> > ---
> > Changes in v2:
> > - Use reStructureText note box for describing the
> >   HBLANK/VBLANK/PIXEL_RATE controls
> > - Document the blanking formula for both CCS and non-CCS raw
> >   sensor drivers
> > - Drop driver-api documentation as common raw sensor model is coming
> >   soon
> > - Link to v1: https://lore.kernel.org/r/20260219-media-fps-docs-v1-1-0387c5c5368c@xxxxxxxxxxxxxxxx
> > ---
> >  .../userspace-api/media/drivers/camera-sensor.rst  | 80 ++++++++++++++++------
> >  1 file changed, 58 insertions(+), 22 deletions(-)
> >
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index 75fd9166383fdbb2dabdb6384ed0904c4e78a3c6..e41ed4616593735bf78b73c2ed34b016ab46c783 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -49,34 +49,70 @@ depends on the type of the device.
> >  Raw camera sensors
> >  ~~~~~~~~~~~~~~~~~~
> >
> > -Instead of a high level parameter such as frame interval, the frame interval is
> > -a result of the configuration of a number of camera sensor implementation
> > -specific parameters. Luckily, these parameters tend to be the same for more or
> > -less all modern raw camera sensors.
> > +Instead of a high level parameter such as frame interval, the frame interval on
> > +a raw camera sensor is determined by a number of sensor-specific parameters.
> > +These parameters tend to be common across most modern raw camera sensors.
> >
> > -The frame interval is calculated using the following equation::
> > +The pixel array is the full grid of photosensitive elements on the camera
> > +sensor. A subregion of it is selected by the analogue crop. The cropped image
> > +may then be subject to binning (averaging of a NxN block) and subsampling which
> > +further reduce the image dimensions. The resulting image is then read out by
> > +the ADC (analogue-to-digital converter) line by line. After ADC readout,
> > +optional digital crop or scaling may further reduce the image dimensions, see
> > +:ref:`VIDIOC_SUBDEV_G_SELECTION <VIDIOC_SUBDEV_G_SELECTION>`.
> >
> > -	frame interval = (analogue crop width + horizontal blanking) *
> > -			 (analogue crop height + vertical blanking) / pixel rate
> > +The frame size is determined by two timing parameters: line length in pixels
> > +(LLP) and frame length in lines (FLL). These are fundamental sensor timing
> > +registers that control how fast the ADC reads out the image. They may go
> 
> nitpicking: I would slightly change this to:
> 
> The frame duration is determined by two timing parameters: line length in pixels
> (LLP) and frame length in lines (FLL). These are fundamental sensor timing
> parameters that control how fast the image is read out.
> 
> 
> > +by different names for a particular sensor, like HMAX and VMAX, or HTOTAL and
> > +VTOTAL, or similar.
> >
> > -The formula is bus independent and is applicable for raw timing parameters on
> > -large variety of devices beyond camera sensors. Devices that have no analogue
> > -crop, use the full source image size, i.e. pixel array size.
> > +LLP is the total number of pixel clock cycles per line, including both the
> > +active readout width and horizontal blanking. FLL is the total number of lines
> > +per frame, including both the active readout height and vertical blanking.
> >
> > -Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and
> > -``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control
> > -is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
> > -the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
> > -sub-device. The unit of that control is pixels per second.
> > +The frame interval is::
> >
> > -Register list-based drivers need to implement read-only sub-device nodes for the
> > -purpose. Devices that are not register list based need these to configure the
> > -device's internal processing pipeline.
> > +        frame interval = (line length in pixels) *
> > +                         (frame length in lines) / pixel rate
> >
> > -The first entity in the linear pipeline is the pixel array. The pixel array may
> > -be followed by other entities that are there to allow configuring binning,
> > -skipping, scaling or digital crop, see :ref:`VIDIOC_SUBDEV_G_SELECTION
> > -<VIDIOC_SUBDEV_G_SELECTION>`.
> > +Application developers can calculate the frame interval using the pixel rate
> > +and blanking controls, relative to the reference rectangle used by the sensor
> > +driver::
> > +
> > +        frame interval = (width + horizontal blanking) *
> > +                         (height + vertical blanking) / pixel rate
> 
> As you below provides definitions for LLP and FLL in the CCS and
> non-CCS case, should we use:
> 
>         frame interval = LLP * FFL / pixel rate
> 
> ?
> 
> > +
> > +.. note::
> > +
> > +        Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK``
> > +        and ``V4L2_CID_VBLANK``, respectively. The unit of the
> > +        ``V4L2_CID_HBLANK`` control is pixels and the unit of the
> > +        ``V4L2_CID_VBLANK`` is lines. The pixel rate in the sensor's **pixel
> > +        array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same sub-device.
> > +        The unit of that control is pixels per second.
> 
> Should we link to those control definitions ?
> 
> > +
> > +        Drivers need to implement sub-device nodes that expose these controls.
> > +        They can be read-only or configurable depending on the device.
> > +
> > +        For non-CCS sensors, the blanking is defined relative to the size of
> > +        the image being sent out to the host over the bus (source pad format)::
> > +
> > +                LLP = output width + horizontal blanking
> > +                FLL = output height + vertical blanking
> > +
> > +        For CCS-compliant raw sensors (that use the CCS driver), the blanking
> > +        controls are defined relative to the analogue crop rectangle::
> > +
> > +                LLP = analogue crop width + horizontal blanking
> > +                FLL = analogue crop height + vertical blanking
> > +
> > +Sensors may achieve higher framerates by allowing lower minimums for line and
> > +frame length when binning or subsampling is enabled.
> 
> I think what you mean here is that sensors can have different
> HBLANK/VBLANK min/max when binning is in use ?
> 
> In that case I would say
> 
> Sensors may allow lower minimums for line and frame length when
> binning or subsampling is enabled.
> 
> 
> > +
> > +The driver shall set the minimum and maximum values of the blanking controls
> 
> and here use "update" instead of "set" to convey that when
> binning/subsampling is in use the blankings valid range could be
> changed if the sensor supports that.
> 
> > +such that the resulting LLP and FLL reigsters are programmed within the range
> > +permitted by the sensor hardware for the current mode.
> 
> Nits apart the patch looks like a good clarification to me.
> 
> Reviewed-by: Jacopo Mondi <jacopo.mondi@xxxxxxxxxxxxxxxx>
> 
> Sakari, I understand you're planning to introduce LLP/FLL controls in
> the generic line-based metadata series.
> 
> Should we take this patch in that series or can this be fast-tracked ?
> I'm asking because if you plan to introduce those controls it might be
> worth mentioning them in this patch from the very beginning.

I think we could merge it before the series; there are few conflicts with
it, too.

-- 
Sakari Ailus