Re: [PATCH] media: Documentation: Fix frame interval calculation for raw camera sensors

[Jai Luthra]

Hi Sakari,

Thanks for the review.

Quoting Sakari Ailus (2026-02-19 15:21:10)
> Hi Jai,
> 
> Thanks for the patch.
> 
> On Thu, Feb 19, 2026 at 01:20:50PM +0530, Jai Luthra wrote:
> > The previous frame interval formula used analogue crop dimensions. This
> > breaks down for some sensors when binning.
> > 
> > For example in imx219 the minimum FLL (frame length in lines) can be
> > lower than the analogue crop height when binning, which would require a
> > negative VBLANK to represent the actual timing. Similarly, imx283 allows
> > a lower minimum HMAX (line length) when doing 2x2 or 3x3 binning than
> > the analogue crop width of the full resolution mode.
> 
> V4L2 integer controls are signed so using negative numbers is a non-issue.
> CCS already does this in some cases actually.
> 

Ah, I see. Control being negative was not the only problem, as it would
also diverge from what sensors and applications already do, but I see you
agreed on that below :-)

I'll update the commit message to make this clear(er).

> > 
> > The CCS specification also describes under section "8.2.6 Line Length
> > and Frame Length" how the horizontal and vertical readout minimums can
> > be different when binning.
> > 
> > Replace the formula with the underlying hardware concepts of LLP (line
> > length in pixels) and FLL (frame length in lines). These terms were
> > chosen to match the CCS specification on raw sensors, as it is a cleaner
> > reference compared to a typical sensor vendor datasheet.
> > 
> > Finally, define the blanking controls relative to the active pixel
> > readout (post-binning) rather than the analogue crop size. This matches
> > what most sensor drivers already do, and also what applications like
> > libcamera expect. In "Figure 42" of CCS specification too, we see a
> > similar definition:
> > 
> >   frame interval = (output width + HBLANK) *
> >                    (output height + VBLANK) / pixel rate
> > 
> > Also add a note in the "Writing camera sensor drivers" guide, to ensure
> > this formula is followed by new sensor drivers.
> 
> We're about to require implementing the common raw sensor model soon by
> essentially all new drivers so documenting the soon-to-be-obsolete state
> has limited benefits.

Makes sense.

> 
> But the UAPI documentation remains relevant for quite some time, please see
> my comments below.
> 
> > 
> > Signed-off-by: Jai Luthra <jai.luthra@xxxxxxxxxxxxxxxx>
> > ---
> >  Documentation/driver-api/media/camera-sensor.rst   | 11 ++++
> >  .../userspace-api/media/drivers/camera-sensor.rst  | 59 +++++++++++++++-------
> >  2 files changed, 53 insertions(+), 17 deletions(-)
> > 
> > diff --git a/Documentation/driver-api/media/camera-sensor.rst b/Documentation/driver-api/media/camera-sensor.rst
> > index 94bd1dae82d5c570b2d11c7faee20dd45d2f4be6..8dcac7551f54ac4ffa71173281ae3bbea331c036 100644
> > --- a/Documentation/driver-api/media/camera-sensor.rst
> > +++ b/Documentation/driver-api/media/camera-sensor.rst
> > @@ -120,6 +120,17 @@ The function returns a non-zero value if it succeeded getting the power count or
> >  runtime PM was disabled, in either of which cases the driver may proceed to
> >  access the device.
> >  
> > +Frame interval
> > +--------------
> > +
> > +If a sensor supports cropping or binning, it is the sensor driver's
> > +responsibility to ensure that the frame interval formula (see
> > +:ref:`media_using_camera_sensor_drivers`) remains valid regardless of the
> > +pipeline configuration. The driver shall adjust the minimum and maximum allowed
> > +values of ``V4L2_CID_HBLANK`` and ``V4L2_CID_VBLANK`` as needed when the mode
> > +changes, so that application developers can always rely on the same formula to
> > +calculate the frame interval.
> > +
> >  Rotation, orientation and flipping
> >  ----------------------------------
> >  
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index 75fd9166383fdbb2dabdb6384ed0904c4e78a3c6..30dddea72a12da264fc9c30e37b561c762c09d29 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -49,35 +49,60 @@ depends on the type of the device.
> >  Raw camera sensors
> >  ~~~~~~~~~~~~~~~~~~
> >  
> > -Instead of a high level parameter such as frame interval, the frame interval is
> > -a result of the configuration of a number of camera sensor implementation
> > -specific parameters. Luckily, these parameters tend to be the same for more or
> > -less all modern raw camera sensors.
> > +Instead of a high level parameter such as frame interval, the frame interval on
> > +a raw camera sensor is determined by a number of sensor-specific parameters.
> > +These parameters tend to be common across most modern raw camera sensors.
> >  
> > -The frame interval is calculated using the following equation::
> > +The pixel array is the full grid of photosensitive elements on the sensor. A
> 
> s/sensor/camera sensor/
> 

Will fix.

> > +subregion of it is selected by the analogue crop. The cropped image may then be
> > +subject to binning (averaging of a NxN block) or subsampling which
> > +further reduce the image dimensions. The resulting image is then read out by
> > +the ADC (analogue-to-digital converter) line by line. After ADC readout,
> > +optional digital crop or scaling may further reduce the image dimensions, see
> > +:ref:`VIDIOC_SUBDEV_G_SELECTION <VIDIOC_SUBDEV_G_SELECTION>`.
> >  
> > -     frame interval = (analogue crop width + horizontal blanking) *
> > -                      (analogue crop height + vertical blanking) / pixel rate
> > +The frame size is determined by two timing parameters: line length in pixels
> > +(LLP) and frame length in lines (FLL). These are fundamental sensor timing
> > +registers that control how fast the ADC reads out the image. They may go
> > +by different names for a particular sensor, like HMAX and VMAX, or HTOTAL and
> > +VTOTAL, or similar.
> >  
> > -The formula is bus independent and is applicable for raw timing parameters on
> > -large variety of devices beyond camera sensors. Devices that have no analogue
> > -crop, use the full source image size, i.e. pixel array size.
> > +LLP is the total number of pixel clock cycles per line, including both the
> > +active readout width and horizontal blanking. FLL is the total number of lines
> > +per frame, including both the active readout height and vertical blanking.
> > +
> > +The frame interval is::
> > +
> > +        frame interval = LLP * FLL / pixel rate
> 
> How would this look like if you spell out LLP and FLL? The rest aren't
> abbreviated either.

        frame interval = (line length in pixels) *
                         (frame length in lines) / pixel rate

This reads much easier, thanks for the suggestion, will update it in v2.

> 
> >  
> >  Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and
> >  ``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control
> >  is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
> > -the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
> > -sub-device. The unit of that control is pixels per second.
> > +the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the
> > +same sub-device. The unit of that control is pixels per second.
> > +
> > +The blanking is defined relative to the size of the image being sent out to the
> > +host over the bus (like CSI-2)::
> > +
> > +        LLP = active width + V4L2_CID_HBLANK
> > +        FLL = active height + V4L2_CID_VBLANK
> > +
> > +The driver shall set the minimum and maximum values of ``V4L2_CID_HBLANK`` and
> > +``V4L2_CID_VBLANK`` such that the resulting LLP and FLL values correspond to the
> > +range permitted by the sensor hardware for the current mode. Sensors that
> > +support binning often define a lower minimum for LLP or FLL registers, which
> > +can help achieve higher framerates when binning.
> > +
> > +Application developers can calculate the frame interval using the output
> > +dimensions and the blanking controls::
> > +
> > +        frame interval = (output width + horizontal blanking) *
> > +                         (output height + vertical blanking) / pixel rate
> 
> This is indeed what many presumably non-CCS sensor drivers implement. Are
> there any that would use the crop rectangle (the imx219 doesn't seem to)?
> For user space variance in this area is of course bad.

I am not aware of any, but I haven't checked all drivers.

The ones I checked (IMX219, IMX283, IMX335, OV5647, OV5640), all set:
[h/v]blank = [h/v]total - mode->[width/height]

So IMO updating the documentation to match that is simpler.

> The common raw sensor model introduces two new controls for the purpose so
> we could re-purpose the old VBLANK/HBLANK controls for this -- apart from
> the CCS driver.
> 

I guess that's my final cue to finally take out time and read that series
properly :-)

Thanks,
    Jai

> >  
> >  Register list-based drivers need to implement read-only sub-device nodes for the
> >  purpose. Devices that are not register list based need these to configure the
> >  device's internal processing pipeline.
> >  
> > -The first entity in the linear pipeline is the pixel array. The pixel array may
> > -be followed by other entities that are there to allow configuring binning,
> > -skipping, scaling or digital crop, see :ref:`VIDIOC_SUBDEV_G_SELECTION
> > -<VIDIOC_SUBDEV_G_SELECTION>`.
> > -
> >  USB cameras etc. devices
> >  ~~~~~~~~~~~~~~~~~~~~~~~~
> >  
> > 
> > ---
> > base-commit: 956b9cbd7f156c8672dac94a00de3c6a0939c692
> > change-id: 20260219-media-fps-docs-fd1da722cc38
> > 
> > Best regards,
> 
> -- 
> Kind regards,
> 
> Sakari Ailus