RE: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

From: Wu, Wentong
Date: Sun Feb 19 2023 - 23:00:58 EST

Next message: JeeHeng Sia: "RE: [PATCH 3/3] RISC-V: Add arch functions to support hibernation/suspend-to-disk"
Previous message: Michael Ellerman: "Re: [PATCHv3, resend] powerpc: mm: radix_tlb: rearrange the if-else block"
In reply to: Laurent Pinchart: "Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

> -----Original Message-----
> From: Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx>
> Sent: Friday, February 17, 2023 7:12 PM
>
> Hello Wentong,
>
> On Fri, Feb 17, 2023 at 06:20:10AM +0000, Wu, Wentong wrote:
> > On Wednesday, February 15, 2023 5:43 PM, Laurent Pinchart wrote:
> > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> > > > is a companion chip designed to provide secure and low power
> > > > vision capability to IA platforms. IVSC is available in existing
> > > > commercial platforms from multiple OEMs.
> > > >
> > > > The primary use case of IVSC is to bring in context awareness.
> > > > IVSC interfaces directly with the platform main camera sensor via
> > > > a CSI-2 link and processes the image data with the embedded AI
> > > > engine. The detected events are sent over I2C to ISH (Intel Sensor
> > > > Hub) for additional data fusion from multiple sensors. The fusion
> > > > results are used to implement advanced use cases like:
> > > > - Face detection to unlock screen
> > > > - Detect user presence to manage backlight setting or waking up
> > > > system
> > >
> > > Do you have plan to support these features in the ivsc driver in the future ?
> >
> > Not sure, but the first step is to upstream this driver.
>
> Sure, no problem.
>
> > > > Since the Image Processing Unit(IPU) used on the host processor
> > > > needs to configure the CSI-2 link in normal camera usages, the
> > > > CSI-2 link and camera sensor can only be used in
> > > > mutually-exclusive ways by host IPU and IVSC. By default the IVSC
> > > > owns the CSI-2 link and camera sensor. The IPU driver can take
> > > > ownership of the CSI-2 link and camera sensor using interfaces provided by
> this IVSC driver.
> > > >
> > > > Switching ownership requires an interface with two different
> > > > hardware modules inside IVSC. The software interface to these
> > > > modules is via Intel MEI (The Intel Management Engine) commands.
> > > > These two hardware modules have two different MEI UUIDs to
> > > > enumerate. These hardware modules are:
> > > > - ACE (Algorithm Context Engine): This module is for algorithm
> > > > computing when IVSC owns camera sensor. Also ACE module controls
> > > > camera sensor's ownership. This hardware module is used to set
> > > > ownership of camera sensor.
> > > > - CSI (Camera Serial Interface): This module is used to route
> > > > camera sensor data either to IVSC or to host for IPU driver and application.
> > > >
> > > > IVSC also provides a privacy mode. When privacy mode is turned on,
> > > > camera sensor can't be used. This means that both ACE and host IPU
> > > > can't get image data. And when this mode is turned on, host IPU
> > > > driver is informed via a registered callback, so that user can be notified.
> > >
> > > How does the privacy mode work, and how can the user trust that the
> > > closed- source IVSC and IME firmwares will honour the privacy settings ?
> >
> > No camera data will be allowed to go through IVSC, and then there will
> > be no data on IVSC CSI transmitter side.
>
> But how can I be sure that the IVSC will not use the camera behind my back, if
> it's all controlled through a closed-source firmware ?

Actually I don't know how to answer your question, but this is guaranteed though we have no plan to open-source the firmware.

>
> > > > In summary, to acquire ownership of camera by IPU driver, first
> > > > ACE module needs to be informed of ownership and then to setup
> > > > MIPI CSI-2 link for the camera sensor and IPU.
> > > >
> > > > Implementation:
> > > > There are two different drivers to handle ACE and CSI hardware
> > > > modules inside IVSC.
> > > > - mei_csi: MEI client driver to send commands and receive notifications
> from CSI module.
> > > > - mei_ace: MEI client driver to send commands and get status from ACE
> module.
> > > > Interface is exposed via ivsc.h to acquire and release camera
> > > > sensor and
> > > > CSI-2 link.
> > >
> > > Do I understand correctly, from your diagram below, that the
> > > communication between the IME and IVSC goes through SPI ?
> > >
> > > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> > > > ------------------------------------------------------------------
> > > > -----------
> > > > | Host Processor |
> > > > | |
> > > > | ----------------- ----------------- --------------- |
> > > > | | | | | | | I2C |
> > > > | | IPU | | ISH | |camera driver|--| |
> > > > | | | | | | | | |
> > > > | ----------------- ----------------- --------------- | |
> > > > | | | | | |
> > > > | | | --------------- | |
> > > > | | | | | | |
> > > > | | | | IVSC driver | | |
> > > > | | | | | | |
> > > > | | | --------------- | |
> > > > | | | | | |
> > > > ----------------|-----------------------|----------------------|---------|---
> > > > | CSI | I2C |SPI |
> > > > | | | |
> > > > ----------------|-----------------------|---------------- | |
> > > > | IVSC | | | |
> > > > | | | | |
> > > > | ----------------- ----------------- | | |
> > > > | | | | | | | |
> > > > | | CSI | | ACE | |------| |
> > > > | | | | | | |
> > > > | ----------------- ----------------- | |
> > > > | | | I2C | |
> > > > ----------------|-----------------------|---------------- |
> > > > | CSI | |
> > > > | | |
> > > > -------------------------------- |
> > > > | | I2C |
> > > > | camera sensor |-----------------------------|
> > > > | |
> > > > --------------------------------
> > > >
> > > > Wentong Wu (3):
> > > > media: pci: intel: ivsc: Add CSI submodule
> > > > media: pci: intel: ivsc: Add ACE submodule
> > > > media: pci: intel: ivsc: Add acquire/release API for ivsc
> > > >
> > > > drivers/media/pci/Kconfig | 1 +
> > > > drivers/media/pci/intel/Makefile | 2 +
> > > > drivers/media/pci/intel/ivsc/Kconfig | 12 +
> > > > drivers/media/pci/intel/ivsc/Makefile | 7 +
> > > > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> > > > drivers/media/pci/intel/ivsc/mei_ace.c | 472
> > > > +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h
> > > > |> 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342
> > > > ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> > > > include/linux/ivsc.h | 74 ++++
> > > > 10 files changed, 1090 insertions(+) create mode 100644
> > > > drivers/media/pci/intel/ivsc/Kconfig
> > > > create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> > > > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> > > > create mode 100644 include/linux/ivsc.h
>
> --
> Regards,
>
> Laurent Pinchart

Next message: JeeHeng Sia: "RE: [PATCH 3/3] RISC-V: Add arch functions to support hibernation/suspend-to-disk"
Previous message: Michael Ellerman: "Re: [PATCHv3, resend] powerpc: mm: radix_tlb: rearrange the if-else block"
In reply to: Laurent Pinchart: "Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]