Re: [PATCH v7 0/6] Add StarFive Camera Subsystem driver

[Jack Zhu]

On 2023/6/19 19:28, Jack Zhu wrote:
> Hi,
> 
> This series is the v7 series that attempts to support the Camera Subsystem
> found on StarFive JH7110 SoC.
> 

Hi everyone,

Could you please help to review and give your comments? 
Thanks a lot!

> The following are the media graph for the device and the v4l2-compliance
> output.
> 
> ===========================================================================
> [the media graph]:
> 
> digraph board {
> 	rankdir=TB
> 	n00000001 [label="{{<port0> 0} | stf_isp0\n/dev/v4l-subdev0 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> 	n00000001:port1 -> n0000000d:port0 [style=dashed]
> 	n00000004 [label="{{<port0> 0} | stf_vin0_wr\n/dev/v4l-subdev1 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> 	n00000004:port1 -> n00000007 [style=bold]
> 	n00000007 [label="stf_vin0_wr_video0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
> 	n0000000d [label="{{<port0> 0} | stf_vin0_isp0\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green]
> 	n0000000d:port1 -> n00000010 [style=bold]
> 	n00000010 [label="stf_vin0_isp0_video1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
> 	n00000018 [label="{{<port0> 0} | cdns_csi2rx.19800000.csi-bridge\n | {<port1> 1 | <port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
> 	n00000018:port1 -> n00000004:port0 [style=dashed]
> 	n00000018:port1 -> n00000001:port0 [style=dashed]
> 	n00000022 [label="{{} | imx219 6-0010\n/dev/v4l-subdev3 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
> 	n00000022:port0 -> n00000018:port0 [style=bold]
> }
> 
> [the device topology]:
> 
> Media controller API version 6.4.0
> 
> Media device information
> ------------------------
> driver          starfive-camss
> model           Starfive Camera Subsystem
> serial          
> bus info        platform:19840000.camss
> hw revision     0x0
> driver version  6.4.0
> 
> Device topology
> - entity 1: stf_isp0 (2 pads, 2 links)
>             type V4L2 subdev subtype Unknown flags 0
>             device node name /dev/v4l-subdev0
> 	pad0: Sink
> 		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb
> 		 crop.bounds:(0,0)/1920x1080
> 		 crop:(0,0)/1920x1080]
> 		<- "cdns_csi2rx.19800000.csi-bridge":1 []
> 	pad1: Source
> 		[fmt:Y12_1X12/1920x1080 field:none colorspace:srgb
> 		 crop.bounds:(0,0)/1920x1080
> 		 crop:(0,0)/1920x1080]
> 		-> "stf_vin0_isp0":0 []
> 
> - entity 4: stf_vin0_wr (2 pads, 2 links)
>             type V4L2 subdev subtype Unknown flags 0
>             device node name /dev/v4l-subdev1
> 	pad0: Sink
> 		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
> 		<- "cdns_csi2rx.19800000.csi-bridge":1 []
> 	pad1: Source
> 		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
> 		-> "stf_vin0_wr_video0":0 [ENABLED,IMMUTABLE]
> 
> - entity 7: stf_vin0_wr_video0 (1 pad, 1 link)
>             type Node subtype V4L flags 0
>             device node name /dev/video0
> 	pad0: Sink
> 		<- "stf_vin0_wr":1 [ENABLED,IMMUTABLE]
> 
> - entity 13: stf_vin0_isp0 (2 pads, 2 links)
>              type V4L2 subdev subtype Unknown flags 0
>              device node name /dev/v4l-subdev2
> 	pad0: Sink
> 		[fmt:Y12_1X12/1920x1080 field:none colorspace:srgb]
> 		<- "stf_isp0":1 []
> 	pad1: Source
> 		[fmt:Y12_1X12/1920x1080 field:none colorspace:srgb]
> 		-> "stf_vin0_isp0_video1":0 [ENABLED,IMMUTABLE]
> 
> - entity 16: stf_vin0_isp0_video1 (1 pad, 1 link)
>              type Node subtype V4L flags 0
>              device node name /dev/video1
> 	pad0: Sink
> 		<- "stf_vin0_isp0":1 [ENABLED,IMMUTABLE]
> 
> - entity 24: cdns_csi2rx.19800000.csi-bridge (5 pads, 3 links)
>              type V4L2 subdev subtype Unknown flags 0
> 	pad0: Sink
> 		<- "imx219 6-0010":0 [ENABLED,IMMUTABLE]
> 	pad1: Source
> 		-> "stf_vin0_wr":0 []
> 		-> "stf_isp0":0 []
> 	pad2: Source
> 	pad3: Source
> 	pad4: Source
> 
> - entity 34: imx219 6-0010 (1 pad, 1 link)
>              type V4L2 subdev subtype Sensor flags 0
>              device node name /dev/v4l-subdev3
> 	pad0: Source
> 		[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
> 		 crop.bounds:(8,8)/3280x2464
> 		 crop:(8,8)/3280x2464]
> 		-> "cdns_csi2rx.19800000.csi-bridge":0 [ENABLED,IMMUTABLE]
> 
> ===========================================================================
> [the v4l2-compliance output]:
> 
> v4l2-compliance -s -d /dev/video1
> v4l2-compliance 1.24.1, 64 bits, 64-bit time_t
> 
> Compliance test for stf camss device /dev/video1:
> 
> Driver Info:
> 	Driver name      : stf camss
> 	Card type        : Starfive Camera Subsystem
> 	Bus info         : platform:19840000.camss
> 	Driver version   : 6.4.0
> 	Capabilities     : 0x85200001
> 		Video Capture
> 		Read/Write
> 		Streaming
> 		Extended Pix Format
> 		Device Capabilities
> 	Device Caps      : 0x05200001
> 		Video Capture
> 		Read/Write
> 		Streaming
> 		Extended Pix Format
> Media Driver Info:
> 	Driver name      : starfive-camss
> 	Model            : Starfive Camera Subsystem
> 	Serial           : 
> 	Bus info         : platform:19840000.camss
> 	Media version    : 6.4.0
> 	Hardware revision: 0x00000000 (0)
> 	Driver version   : 6.4.0
> Interface Info:
> 	ID               : 0x03000012
> 	Type             : V4L Video
> Entity Info:
> 	ID               : 0x00000010 (16)
> 	Name             : stf_vin0_isp0_video1
> 	Function         : V4L2 I/O
> 	Pad 0x01000011   : 0: Sink
> 	  Link 0x02000014: from remote pad 0x100000f of entity 'stf_vin0_isp0' (Video Pixel Formatter): Data, Enabled, Immutable
> 
> Required ioctls:
> 	test MC information (see 'Media Driver Info' above): OK
> 	test VIDIOC_QUERYCAP: OK
> 	test invalid ioctls: OK
> 
> Allow for multiple opens:
> 	test second /dev/video1 open: OK
> 	test VIDIOC_QUERYCAP: OK
> 	test VIDIOC_G/S_PRIORITY: OK
> 	test for unlimited opens: OK
> 
> Debug ioctls:
> 	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
> 	test VIDIOC_LOG_STATUS: OK (Not Supported)
> 
> Input ioctls:
> 	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
> 	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> 	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
> 	test VIDIOC_ENUMAUDIO: OK (Not Supported)
> 	test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
> 	test VIDIOC_G/S_AUDIO: OK (Not Supported)
> 	Inputs: 0 Audio Inputs: 0 Tuners: 0
> 
> Output ioctls:
> 	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
> 	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
> 	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
> 	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
> 	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
> 	Outputs: 0 Audio Outputs: 0 Modulators: 0
> 
> Input/Output configuration ioctls:
> 	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
> 	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
> 	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
> 	test VIDIOC_G/S_EDID: OK (Not Supported)
> 
> Control ioctls:
> 	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
> 	test VIDIOC_QUERYCTRL: OK (Not Supported)
> 	test VIDIOC_G/S_CTRL: OK (Not Supported)
> 	test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
> 	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
> 	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
> 	Standard Controls: 0 Private Controls: 0
> 
> Format ioctls:
> 	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
> 	test VIDIOC_G/S_PARM: OK (Not Supported)
> 	test VIDIOC_G_FBUF: OK (Not Supported)
> 	test VIDIOC_G_FMT: OK
> 	test VIDIOC_TRY_FMT: OK
> 	test VIDIOC_S_FMT: OK
> 	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
> 	test Cropping: OK (Not Supported)
> 	test Composing: OK (Not Supported)
> 	test Scaling: OK
> 
> Codec ioctls:
> 	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
> 	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
> 	test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
> 
> Buffer ioctls:
> 	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
> 	test VIDIOC_EXPBUF: OK
> 	test Requests: OK (Not Supported)
> 
> Test input 0:
> 
> Streaming ioctls:
> 	test read/write: OK
> 	test blocking wait: OK
> 	test MMAP (no poll): OK                           
> 	test MMAP (select): OK                            
> 	test MMAP (epoll): OK                             
> 	test USERPTR (no poll): OK (Not Supported)
> 	test USERPTR (select): OK (Not Supported)
> 	test DMABUF: Cannot test, specify --expbuf-device
> 
> Total for stf camss device /dev/video1: 53, Succeeded: 53, Failed: 0, Warnings: 0
> 
> ===========================================================================
> 
> Changes in v7:
> - HAS_DMA is used instead of DMA_CMA in Kconfig.
> - Dropped some non-essential member variables.
> - Used v4l2_async_nf_add_fwnode_remote() to simplify the relevant code.
> - Modified some Local variable types in the function.
> - Used v4l2_create_fwnode_links_to_pad() to simplify the relevant code.
> - Added error handling for clk_prepare_enable().
> - Simplified stfcamss_format_info struct and modified the relevant code.
> - Dropped enum_input, g_input and s_input.
> - Unified v4l2_ioctl_ops struct.
> - Used v4l2_fh_open()/vb2_fop_release to replace deprecated APIs.
> - Added a camss directory under the starfive directory and modified the
>   patch title.
> 
> v6 link: https://lore.kernel.org/all/20230525083202.67933-1-jack.zhu@xxxxxxxxxxxxxxxx/
> 
> Changes in v6:
> - Added 'bus-type' in bindings example.
> - Corrected spelling errors.
> - As reviewed by Bryan, used 'nclks' and 'nrsts' variables.
> - Added lccf config for ISP.
> 
> v5 link: https://lore.kernel.org/all/20230512102844.51084-1-jack.zhu@xxxxxxxxxxxxxxxx/
> 
> Changes in v5:
> - Rebased on v6.4-rc1.
> - Added new patch.
> - Modified ISP driver.
> 
> v4 link: https://lore.kernel.org/all/20230413035541.62129-1-jack.zhu@xxxxxxxxxxxxxxxx/
> 
> Previous cover letter from v4:
> 
> This patch series adds support for the StarFive Camera Subsystem
> found on StarFive JH7110 SoC.
> 
> The driver implements V4L2, Media controller and V4L2 subdev interfaces.
> Camera sensor using V4L2 subdev interface in the kernel is supported.
> 
> The driver is tested on VisionFive V2 board with IMX219 camera sensor.
> GStreamer 1.18.5 with v4l2src plugin is supported.
> 
> Changes since v3:
> Patch 1:
> - Modified port@0 and port@1 properties.
> - Extended the port@0 example with appropriate properties.
> - Added 'port@0' for 'required'
> Patch 2:
> - Modified spelling errors.
> Patch 3:
> - Merged patch 5 into the patch with an explanation for compatible in
>   commit msg.
> Patch 6:
> - Asserted pixel_rst[i] reset in the loop after the err_disable_pixclk
>   label.
> - Modified Code Style for getting sys_rst and p_rst.
> - Renamed clk_name to name and modified the relevant code.
> Patch 9:
> - Added static for stfcamss_get_mem_res function.
> - Added static for isp_close function.
> - Fixed implicit conversion warning for stf_vin_map_isp_pad function.
> - Dropped unused variables.
> 
>   v3: https://lore.kernel.org/all/20230331121826.96973-1-jack.zhu@xxxxxxxxxxxxxxxx/
> 
> Changes since v2:
> - Rebased on v6.3-rc1.
> Patch 1:
> - Modified spelling errors.
> - Added port@0.
> - Modified '$ref' of port.
> - Added 'ports' to 'required'.
> - Dropped 'stfcamss' label in example.
> - Added port@0 in example.
> - Added MAINTAINERS file.
> Patch 2:
> - Split this patch into three new patches.
> - Modified compatible property.
> - Replaced clock names with the existing names.
> - Modified 'bus-type' and 'clock-lanes'
> - Added port@2 - port@4
> - Dropped 'csi2rx' label in example.
> Patch 3:
> - Updated rst and dot file as three pipelines were deleted.
> Patch 4:
> - Split this patch into three new patches.
> - Dropped .s_power() and .get_fmt().
> - Dropped CSI-2 DT support.
> - Dropped v4l2_device_register_subdev_nodes().
> - Used assigned-clock-rates in DT to set clk value.
> - Modified 'compatible' field.
> Patch 5:
> - Deleted three pipelines. 
> - Modified 'stfcamss_clocks'/'stfcamss_resets' struct.
> - Dropped stfcamss_find_sensor() function.
> - Removed redundant code from stfcamss_of_parse_endpoint_node().
> - Modified spelling errors.
> - Rewrote stfcamss_reg_media_subdev_node() function.
> - Modified stfcamss_subdev_notifier_bound().
> - Modified stfcamss_probe() function.
> - Dropped stfcamss_suspend() and stfcamss_resume().
> - Dropped dev_info() in stfcamss_remove() function.
> - Added 'stf_' prefix for enum subdev_type.
> - Moved all includes to the top in stf_camss.h file.
> - Dropped unused fields in stfcamss struct.
> - Replaced Custom logging macros with regular macros.
> - Rewrote register read and write functions.
> - Used lowercase for all hex constants.
> - Used macro to name registers.
> - Dropped unused ioctl and stf_isp_ioctl.h file.
> 
>   v2: https://lore.kernel.org/all/20230310120553.60586-1-jack.zhu@xxxxxxxxxxxxxxxx/
> 
> Changes since v1:
> - Deleted starfive,jh7110-mipi-csi2.yaml.
> - Converted cdns,csi2rx.txt to cdns,csi2rx.yaml and added ‘resets’
>   properties.
> - Added ‘cdns,csi2rx.yaml’ in ‘CADENCE MIPI-CSI2 BRIDGES’ entry.
> - The following contents were modified in starfive,jh7110-camss.yaml:
>   dropped quotes from ’id’ and ‘schema’; dropped ‘|’ for ‘description’;
>   corrected the wrong or redundant words: ‘a ISP’, ‘PD ISP’;
>   dropped ‘minItems’ for ‘reg’, ‘clocks’, ‘resets’ and ‘interrupts’;
>   dropped the '_clk' and 'rst_' prefix about the 'clock-names' and
>   'reset-names';
>   changed ‘endpoint@1’ to ‘endpoint’; updated examples;
> - Updated Subject for some patches.
> - Merged patch 6, 7, 8, 9, 10, 11 into one patch.
> 
> Jack Zhu (6):
>   media: dt-bindings: Add JH7110 Camera Subsystem
>   media: admin-guide: Add starfive_camss.rst for Starfive Camera
>     Subsystem
>   media: starfive: camss: Add basic driver
>   media: starfive: camss: Add video driver
>   media: starfive: camss: Add ISP driver
>   media: starfive: camss: Add VIN driver
> 
>  .../admin-guide/media/starfive_camss.rst      |   57 +
>  .../media/starfive_camss_graph.dot            |   16 +
>  .../admin-guide/media/v4l-drivers.rst         |    1 +
>  .../bindings/media/starfive,jh7110-camss.yaml |  180 +++
>  MAINTAINERS                                   |    9 +
>  drivers/media/platform/Kconfig                |    1 +
>  drivers/media/platform/Makefile               |    1 +
>  drivers/media/platform/starfive/Kconfig       |    5 +
>  drivers/media/platform/starfive/Makefile      |    2 +
>  drivers/media/platform/starfive/camss/Kconfig |   16 +
>  .../media/platform/starfive/camss/Makefile    |   14 +
>  .../media/platform/starfive/camss/stf_camss.c |  444 +++++++
>  .../media/platform/starfive/camss/stf_camss.h |  151 +++
>  .../media/platform/starfive/camss/stf_isp.c   |  519 ++++++++
>  .../media/platform/starfive/camss/stf_isp.h   |  479 ++++++++
>  .../platform/starfive/camss/stf_isp_hw_ops.c  |  468 ++++++++
>  .../media/platform/starfive/camss/stf_video.c |  724 +++++++++++
>  .../media/platform/starfive/camss/stf_video.h |   92 ++
>  .../media/platform/starfive/camss/stf_vin.c   | 1069 +++++++++++++++++
>  .../media/platform/starfive/camss/stf_vin.h   |  173 +++
>  .../platform/starfive/camss/stf_vin_hw_ops.c  |  192 +++
>  21 files changed, 4613 insertions(+)
>  create mode 100644 Documentation/admin-guide/media/starfive_camss.rst
>  create mode 100644 Documentation/admin-guide/media/starfive_camss_graph.dot
>  create mode 100644 Documentation/devicetree/bindings/media/starfive,jh7110-camss.yaml
>  create mode 100644 drivers/media/platform/starfive/Kconfig
>  create mode 100644 drivers/media/platform/starfive/Makefile
>  create mode 100644 drivers/media/platform/starfive/camss/Kconfig
>  create mode 100644 drivers/media/platform/starfive/camss/Makefile
>  create mode 100644 drivers/media/platform/starfive/camss/stf_camss.c
>  create mode 100644 drivers/media/platform/starfive/camss/stf_camss.h
>  create mode 100644 drivers/media/platform/starfive/camss/stf_isp.c
>  create mode 100644 drivers/media/platform/starfive/camss/stf_isp.h
>  create mode 100644 drivers/media/platform/starfive/camss/stf_isp_hw_ops.c
>  create mode 100644 drivers/media/platform/starfive/camss/stf_video.c
>  create mode 100644 drivers/media/platform/starfive/camss/stf_video.h
>  create mode 100644 drivers/media/platform/starfive/camss/stf_vin.c
>  create mode 100644 drivers/media/platform/starfive/camss/stf_vin.h
>  create mode 100644 drivers/media/platform/starfive/camss/stf_vin_hw_ops.c
> 

-- 
Regards,

Jack Zhu