Re: [PATCH] media: Documentation: Fix frame interval calculation for raw camera sensors

[Sakari Ailus]

Hi Jacopo,

On Mon, Feb 23, 2026 at 10:11:02AM +0100, Jacopo Mondi wrote:
> Hi Jai
> 
> On Thu, Feb 19, 2026 at 01:20:50PM +0530, Jai Luthra wrote:
> > The previous frame interval formula used analogue crop dimensions. This
> > breaks down for some sensors when binning.
> >
> > For example in imx219 the minimum FLL (frame length in lines) can be
> > lower than the analogue crop height when binning, which would require a
> > negative VBLANK to represent the actual timing. Similarly, imx283 allows
> > a lower minimum HMAX (line length) when doing 2x2 or 3x3 binning than
> > the analogue crop width of the full resolution mode.
> >
> > The CCS specification also describes under section "8.2.6 Line Length
> > and Frame Length" how the horizontal and vertical readout minimums can
> > be different when binning.
> >
> > Replace the formula with the underlying hardware concepts of LLP (line
> > length in pixels) and FLL (frame length in lines). These terms were
> > chosen to match the CCS specification on raw sensors, as it is a cleaner
> > reference compared to a typical sensor vendor datasheet.
> >
> > Finally, define the blanking controls relative to the active pixel
> > readout (post-binning) rather than the analogue crop size. This matches
> > what most sensor drivers already do, and also what applications like
> > libcamera expect. In "Figure 42" of CCS specification too, we see a
> > similar definition:
> >
> >   frame interval = (output width + HBLANK) *
> >                    (output height + VBLANK) / pixel rate
> >
> > Also add a note in the "Writing camera sensor drivers" guide, to ensure
> > this formula is followed by new sensor drivers.
> 
> I agree that using the analogue crop rectangle sizes is not correct,
> however, with this new formulation the LLP and FLL values might be
> smaller than the crop rectangles reported through the selection API,
> which I'm not sure it's great from a consistency point of view ?
> 
> Also the below suggested formulation:
> 
>         LLP = active width + V4L2_CID_HBLANK
>         FLL = active height + V4L2_CID_VBLANK
> 
> Assumes the combined effect of [binning + subsampling] always directly
> affect the readout time of pixels on the pixel array. The CCS specs
> and driver seems to suggest that's not always the case ?
> 
> Do we need to distinguish between binning modes that affect the
> timings and binning modes that do not do that?

For existing (pre-common raw sensor model) drivers, I'd say "no". CCS and
common raw sensor model will use line length in pixels and frame length in
lines for this, which reflects what the hardware does.

Existing non-CCS drivers have used the format for this purpose and adjusted
the blanking values accordingly.

> 
> As we're going to introduce a control for binning to report the
> binning factor in the image dimension domain, should we introduce
> a control to specify the binning factor in the image timing domain ?
> 
>         LLP = (analog_crop_width + HBLANK) / binning_timing_h
>         FLL = (analog_crop_height + VBLANK) / binning_timing_v
>         frame_interval = LLP * FLL / pixel_rate
> 
> I'm not 100% sure this is correct however, as the blankings should be
> expressed on a different clock domain that the pixel sampling rate,
> but I guess this is a reasonable approximation ?

If binning is done in analogue domain, the above would hold, but sensor do
digital binning as well. The driver needs to adjust the blanking values
(and in the future frame length and line length values) to accommodate
this.

> 
> >
> > Signed-off-by: Jai Luthra <jai.luthra@xxxxxxxxxxxxxxxx>
> > ---
> >  Documentation/driver-api/media/camera-sensor.rst   | 11 ++++
> >  .../userspace-api/media/drivers/camera-sensor.rst  | 59 +++++++++++++++-------
> >  2 files changed, 53 insertions(+), 17 deletions(-)
> >
> > diff --git a/Documentation/driver-api/media/camera-sensor.rst b/Documentation/driver-api/media/camera-sensor.rst
> > index 94bd1dae82d5c570b2d11c7faee20dd45d2f4be6..8dcac7551f54ac4ffa71173281ae3bbea331c036 100644
> > --- a/Documentation/driver-api/media/camera-sensor.rst
> > +++ b/Documentation/driver-api/media/camera-sensor.rst
> > @@ -120,6 +120,17 @@ The function returns a non-zero value if it succeeded getting the power count or
> >  runtime PM was disabled, in either of which cases the driver may proceed to
> >  access the device.
> >
> > +Frame interval
> > +--------------
> > +
> > +If a sensor supports cropping or binning, it is the sensor driver's
> > +responsibility to ensure that the frame interval formula (see
> > +:ref:`media_using_camera_sensor_drivers`) remains valid regardless of the
> > +pipeline configuration. The driver shall adjust the minimum and maximum allowed
> > +values of ``V4L2_CID_HBLANK`` and ``V4L2_CID_VBLANK`` as needed when the mode
> > +changes, so that application developers can always rely on the same formula to
> > +calculate the frame interval.
> > +
> >  Rotation, orientation and flipping
> >  ----------------------------------
> >
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index 75fd9166383fdbb2dabdb6384ed0904c4e78a3c6..30dddea72a12da264fc9c30e37b561c762c09d29 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -49,35 +49,60 @@ depends on the type of the device.
> >  Raw camera sensors
> >  ~~~~~~~~~~~~~~~~~~
> >
> > -Instead of a high level parameter such as frame interval, the frame interval is
> > -a result of the configuration of a number of camera sensor implementation
> > -specific parameters. Luckily, these parameters tend to be the same for more or
> > -less all modern raw camera sensors.
> > +Instead of a high level parameter such as frame interval, the frame interval on
> > +a raw camera sensor is determined by a number of sensor-specific parameters.
> > +These parameters tend to be common across most modern raw camera sensors.
> >
> > -The frame interval is calculated using the following equation::
> > +The pixel array is the full grid of photosensitive elements on the sensor. A
> > +subregion of it is selected by the analogue crop. The cropped image may then be
> > +subject to binning (averaging of a NxN block) or subsampling which
> > +further reduce the image dimensions. The resulting image is then read out by
> > +the ADC (analogue-to-digital converter) line by line. After ADC readout,
> > +optional digital crop or scaling may further reduce the image dimensions, see
> > +:ref:`VIDIOC_SUBDEV_G_SELECTION <VIDIOC_SUBDEV_G_SELECTION>`.
> >
> > -	frame interval = (analogue crop width + horizontal blanking) *
> > -			 (analogue crop height + vertical blanking) / pixel rate
> > +The frame size is determined by two timing parameters: line length in pixels
> > +(LLP) and frame length in lines (FLL). These are fundamental sensor timing
> > +registers that control how fast the ADC reads out the image. They may go
> > +by different names for a particular sensor, like HMAX and VMAX, or HTOTAL and
> > +VTOTAL, or similar.
> >
> > -The formula is bus independent and is applicable for raw timing parameters on
> > -large variety of devices beyond camera sensors. Devices that have no analogue
> > -crop, use the full source image size, i.e. pixel array size.
> > +LLP is the total number of pixel clock cycles per line, including both the
> > +active readout width and horizontal blanking. FLL is the total number of lines
> > +per frame, including both the active readout height and vertical blanking.
> > +
> > +The frame interval is::
> > +
> > +        frame interval = LLP * FLL / pixel rate
> >
> >  Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and
> >  ``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control
> >  is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
> > -the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
> > -sub-device. The unit of that control is pixels per second.
> > +the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the
> > +same sub-device. The unit of that control is pixels per second.
> > +
> > +The blanking is defined relative to the size of the image being sent out to the
> > +host over the bus (like CSI-2)::
> > +
> > +        LLP = active width + V4L2_CID_HBLANK
> > +        FLL = active height + V4L2_CID_VBLANK
> > +
> > +The driver shall set the minimum and maximum values of ``V4L2_CID_HBLANK`` and
> > +``V4L2_CID_VBLANK`` such that the resulting LLP and FLL values correspond to the
> > +range permitted by the sensor hardware for the current mode. Sensors that
> > +support binning often define a lower minimum for LLP or FLL registers, which
> > +can help achieve higher framerates when binning.
> > +
> > +Application developers can calculate the frame interval using the output
> > +dimensions and the blanking controls::
> > +
> > +        frame interval = (output width + horizontal blanking) *
> > +                         (output height + vertical blanking) / pixel rate
> >
> >  Register list-based drivers need to implement read-only sub-device nodes for the
> >  purpose. Devices that are not register list based need these to configure the
> >  device's internal processing pipeline.
> >
> > -The first entity in the linear pipeline is the pixel array. The pixel array may
> > -be followed by other entities that are there to allow configuring binning,
> > -skipping, scaling or digital crop, see :ref:`VIDIOC_SUBDEV_G_SELECTION
> > -<VIDIOC_SUBDEV_G_SELECTION>`.
> > -
> >  USB cameras etc. devices
> >  ~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >
> > ---
> > base-commit: 956b9cbd7f156c8672dac94a00de3c6a0939c692
> > change-id: 20260219-media-fps-docs-fd1da722cc38
> >
> > Best regards,
> > --
> > Jai Luthra <jai.luthra@xxxxxxxxxxxxxxxx>
> >

-- 
Kind regards,

Sakari Ailus