[PATCH v7 0/6] Add StarFive Camera Subsystem driver

From: Jack Zhu
Date: Mon Jun 19 2023 - 07:29:31 EST

Hi,

This series is the v7 series that attempts to support the Camera Subsystem
found on StarFive JH7110 SoC.

The following are the media graph for the device and the v4l2-compliance
output.

===========================================================================
[the media graph]:

digraph board {
rankdir=TB
n00000001 [label="{{<port0> 0} | stf_isp0\n/dev/v4l-subdev0 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
n00000001:port1 -> n0000000d:port0 [style=dashed]
n00000004 [label="{{<port0> 0} | stf_vin0_wr\n/dev/v4l-subdev1 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
n00000004:port1 -> n00000007 [style=bold]
n00000007 [label="stf_vin0_wr_video0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
n0000000d [label="{{<port0> 0} | stf_vin0_isp0\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
n0000000d:port1 -> n00000010 [style=bold]
n00000010 [label="stf_vin0_isp0_video1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
n00000018 [label="{{<port0> 0} | cdns_csi2rx.19800000.csi-bridge\n | {<port1> 1 | <port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
n00000018:port1 -> n00000004:port0 [style=dashed]
n00000018:port1 -> n00000001:port0 [style=dashed]
n00000022 [label="{{} | imx219 6-0010\n/dev/v4l-subdev3 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
n00000022:port0 -> n00000018:port0 [style=bold]
}

[the device topology]:

Media controller API version 6.4.0

Media device information
------------------------
driver starfive-camss
model Starfive Camera Subsystem
serial
bus info platform:19840000.camss
hw revision 0x0
driver version 6.4.0

Device topology
- entity 1: stf_isp0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb
crop.bounds:(0,0)/1920x1080
crop:(0,0)/1920x1080]
<- "cdns_csi2rx.19800000.csi-bridge":1 []
pad1: Source
[fmt:Y12_1X12/1920x1080 field:none colorspace:srgb
crop.bounds:(0,0)/1920x1080
crop:(0,0)/1920x1080]
-> "stf_vin0_isp0":0 []

- entity 4: stf_vin0_wr (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
<- "cdns_csi2rx.19800000.csi-bridge":1 []
pad1: Source
[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
-> "stf_vin0_wr_video0":0 [ENABLED,IMMUTABLE]

- entity 7: stf_vin0_wr_video0 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Sink
<- "stf_vin0_wr":1 [ENABLED,IMMUTABLE]

- entity 13: stf_vin0_isp0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:Y12_1X12/1920x1080 field:none colorspace:srgb]
<- "stf_isp0":1 []
pad1: Source
[fmt:Y12_1X12/1920x1080 field:none colorspace:srgb]
-> "stf_vin0_isp0_video1":0 [ENABLED,IMMUTABLE]

- entity 16: stf_vin0_isp0_video1 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video1
pad0: Sink
<- "stf_vin0_isp0":1 [ENABLED,IMMUTABLE]

- entity 24: cdns_csi2rx.19800000.csi-bridge (5 pads, 3 links)
type V4L2 subdev subtype Unknown flags 0
pad0: Sink
<- "imx219 6-0010":0 [ENABLED,IMMUTABLE]
pad1: Source
-> "stf_vin0_wr":0 []
-> "stf_isp0":0 []
pad2: Source
pad3: Source
pad4: Source

- entity 34: imx219 6-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(8,8)/3280x2464
crop:(8,8)/3280x2464]
-> "cdns_csi2rx.19800000.csi-bridge":0 [ENABLED,IMMUTABLE]

===========================================================================
[the v4l2-compliance output]:

v4l2-compliance -s -d /dev/video1
v4l2-compliance 1.24.1, 64 bits, 64-bit time_t

Compliance test for stf camss device /dev/video1:

Driver Info:
Driver name : stf camss
Card type : Starfive Camera Subsystem
Bus info : platform:19840000.camss
Driver version : 6.4.0
Capabilities : 0x85200001
Video Capture
Read/Write
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x05200001
Video Capture
Read/Write
Streaming
Extended Pix Format
Media Driver Info:
Driver name : starfive-camss
Model : Starfive Camera Subsystem
Serial :
Bus info : platform:19840000.camss
Media version : 6.4.0
Hardware revision: 0x00000000 (0)
Driver version : 6.4.0
Interface Info:
ID : 0x03000012
Type : V4L Video
Entity Info:
ID : 0x00000010 (16)
Name : stf_vin0_isp0_video1
Function : V4L2 I/O
Pad 0x01000011 : 0: Sink
Link 0x02000014: from remote pad 0x100000f of entity 'stf_vin0_isp0' (Video Pixel Formatter): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK
test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video1 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0

Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK

Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported)

Test input 0:

Streaming ioctls:
test read/write: OK
test blocking wait: OK
test MMAP (no poll): OK
test MMAP (select): OK
test MMAP (epoll): OK
test USERPTR (no poll): OK (Not Supported)
test USERPTR (select): OK (Not Supported)
test DMABUF: Cannot test, specify --expbuf-device

Total for stf camss device /dev/video1: 53, Succeeded: 53, Failed: 0, Warnings: 0

===========================================================================

Changes in v7:
- HAS_DMA is used instead of DMA_CMA in Kconfig.
- Dropped some non-essential member variables.
- Used v4l2_async_nf_add_fwnode_remote() to simplify the relevant code.
- Modified some Local variable types in the function.
- Used v4l2_create_fwnode_links_to_pad() to simplify the relevant code.
- Added error handling for clk_prepare_enable().
- Simplified stfcamss_format_info struct and modified the relevant code.
- Dropped enum_input, g_input and s_input.
- Unified v4l2_ioctl_ops struct.
- Used v4l2_fh_open()/vb2_fop_release to replace deprecated APIs.
- Added a camss directory under the starfive directory and modified the
patch title.

v6 link: https://lore.kernel.org/all/20230525083202.67933-1-jack.zhu@xxxxxxxxxxxxxxxx/

Changes in v6:
- Added 'bus-type' in bindings example.
- Corrected spelling errors.
- As reviewed by Bryan, used 'nclks' and 'nrsts' variables.
- Added lccf config for ISP.

v5 link: https://lore.kernel.org/all/20230512102844.51084-1-jack.zhu@xxxxxxxxxxxxxxxx/

Changes in v5:
- Rebased on v6.4-rc1.
- Added new patch.
- Modified ISP driver.

v4 link: https://lore.kernel.org/all/20230413035541.62129-1-jack.zhu@xxxxxxxxxxxxxxxx/

Previous cover letter from v4:

This patch series adds support for the StarFive Camera Subsystem
found on StarFive JH7110 SoC.

The driver implements V4L2, Media controller and V4L2 subdev interfaces.
Camera sensor using V4L2 subdev interface in the kernel is supported.

The driver is tested on VisionFive V2 board with IMX219 camera sensor.
GStreamer 1.18.5 with v4l2src plugin is supported.

Changes since v3:
Patch 1:
- Modified port@0 and port@1 properties.
- Extended the port@0 example with appropriate properties.
- Added 'port@0' for 'required'
Patch 2:
- Modified spelling errors.
Patch 3:
- Merged patch 5 into the patch with an explanation for compatible in
commit msg.
Patch 6:
- Asserted pixel_rst[i] reset in the loop after the err_disable_pixclk
label.
- Modified Code Style for getting sys_rst and p_rst.
- Renamed clk_name to name and modified the relevant code.
Patch 9:
- Added static for stfcamss_get_mem_res function.
- Added static for isp_close function.
- Fixed implicit conversion warning for stf_vin_map_isp_pad function.
- Dropped unused variables.

v3: https://lore.kernel.org/all/20230331121826.96973-1-jack.zhu@xxxxxxxxxxxxxxxx/

Changes since v2:
- Rebased on v6.3-rc1.
Patch 1:
- Modified spelling errors.
- Added port@0.
- Modified '$ref' of port.
- Added 'ports' to 'required'.
- Dropped 'stfcamss' label in example.
- Added port@0 in example.
- Added MAINTAINERS file.
Patch 2:
- Split this patch into three new patches.
- Modified compatible property.
- Replaced clock names with the existing names.
- Modified 'bus-type' and 'clock-lanes'
- Added port@2 - port@4
- Dropped 'csi2rx' label in example.
Patch 3:
- Updated rst and dot file as three pipelines were deleted.
Patch 4:
- Split this patch into three new patches.
- Dropped .s_power() and .get_fmt().
- Dropped CSI-2 DT support.
- Dropped v4l2_device_register_subdev_nodes().
- Used assigned-clock-rates in DT to set clk value.
- Modified 'compatible' field.
Patch 5:
- Deleted three pipelines.
- Modified 'stfcamss_clocks'/'stfcamss_resets' struct.
- Dropped stfcamss_find_sensor() function.
- Removed redundant code from stfcamss_of_parse_endpoint_node().
- Modified spelling errors.
- Rewrote stfcamss_reg_media_subdev_node() function.
- Modified stfcamss_subdev_notifier_bound().
- Modified stfcamss_probe() function.
- Dropped stfcamss_suspend() and stfcamss_resume().
- Dropped dev_info() in stfcamss_remove() function.
- Added 'stf_' prefix for enum subdev_type.
- Moved all includes to the top in stf_camss.h file.
- Dropped unused fields in stfcamss struct.
- Replaced Custom logging macros with regular macros.
- Rewrote register read and write functions.
- Used lowercase for all hex constants.
- Used macro to name registers.
- Dropped unused ioctl and stf_isp_ioctl.h file.

v2: https://lore.kernel.org/all/20230310120553.60586-1-jack.zhu@xxxxxxxxxxxxxxxx/

Changes since v1:
- Deleted starfive,jh7110-mipi-csi2.yaml.
- Converted cdns,csi2rx.txt to cdns,csi2rx.yaml and added ‘resets’
properties.
- Added ‘cdns,csi2rx.yaml’ in ‘CADENCE MIPI-CSI2 BRIDGES’ entry.
- The following contents were modified in starfive,jh7110-camss.yaml:
dropped quotes from ’id’ and ‘schema’; dropped ‘|’ for ‘description’;
corrected the wrong or redundant words: ‘a ISP’, ‘PD ISP’;
dropped ‘minItems’ for ‘reg’, ‘clocks’, ‘resets’ and ‘interrupts’;
dropped the '_clk' and 'rst_' prefix about the 'clock-names' and
'reset-names';
changed ‘endpoint@1’ to ‘endpoint’; updated examples;
- Updated Subject for some patches.
- Merged patch 6, 7, 8, 9, 10, 11 into one patch.

Jack Zhu (6):
media: dt-bindings: Add JH7110 Camera Subsystem
media: admin-guide: Add starfive_camss.rst for Starfive Camera
Subsystem
media: starfive: camss: Add basic driver
media: starfive: camss: Add video driver
media: starfive: camss: Add ISP driver
media: starfive: camss: Add VIN driver

.../admin-guide/media/starfive_camss.rst | 57 +
.../media/starfive_camss_graph.dot | 16 +
.../admin-guide/media/v4l-drivers.rst | 1 +
.../bindings/media/starfive,jh7110-camss.yaml | 180 +++
MAINTAINERS | 9 +
drivers/media/platform/Kconfig | 1 +
drivers/media/platform/Makefile | 1 +
drivers/media/platform/starfive/Kconfig | 5 +
drivers/media/platform/starfive/Makefile | 2 +
drivers/media/platform/starfive/camss/Kconfig | 16 +
.../media/platform/starfive/camss/Makefile | 14 +
.../media/platform/starfive/camss/stf_camss.c | 444 +++++++
.../media/platform/starfive/camss/stf_camss.h | 151 +++
.../media/platform/starfive/camss/stf_isp.c | 519 ++++++++
.../media/platform/starfive/camss/stf_isp.h | 479 ++++++++
.../platform/starfive/camss/stf_isp_hw_ops.c | 468 ++++++++
.../media/platform/starfive/camss/stf_video.c | 724 +++++++++++
.../media/platform/starfive/camss/stf_video.h | 92 ++
.../media/platform/starfive/camss/stf_vin.c | 1069 +++++++++++++++++
.../media/platform/starfive/camss/stf_vin.h | 173 +++
.../platform/starfive/camss/stf_vin_hw_ops.c | 192 +++
21 files changed, 4613 insertions(+)
create mode 100644 Documentation/admin-guide/media/starfive_camss.rst
create mode 100644 Documentation/admin-guide/media/starfive_camss_graph.dot
create mode 100644 Documentation/devicetree/bindings/media/starfive,jh7110-camss.yaml
create mode 100644 drivers/media/platform/starfive/Kconfig
create mode 100644 drivers/media/platform/starfive/Makefile
create mode 100644 drivers/media/platform/starfive/camss/Kconfig
create mode 100644 drivers/media/platform/starfive/camss/Makefile
create mode 100644 drivers/media/platform/starfive/camss/stf_camss.c
create mode 100644 drivers/media/platform/starfive/camss/stf_camss.h
create mode 100644 drivers/media/platform/starfive/camss/stf_isp.c
create mode 100644 drivers/media/platform/starfive/camss/stf_isp.h
create mode 100644 drivers/media/platform/starfive/camss/stf_isp_hw_ops.c
create mode 100644 drivers/media/platform/starfive/camss/stf_video.c
create mode 100644 drivers/media/platform/starfive/camss/stf_video.h
create mode 100644 drivers/media/platform/starfive/camss/stf_vin.c
create mode 100644 drivers/media/platform/starfive/camss/stf_vin.h
create mode 100644 drivers/media/platform/starfive/camss/stf_vin_hw_ops.c

--
2.34.1