Re: [PATCH 01/22] Add Vision Processing Unit (VPU) documentation.

[mark gross]

On Fri, Dec 18, 2020 at 03:30:18PM -0800, Randy Dunlap wrote:
> Hi--
> 
> On 12/1/20 2:34 PM, mgross@xxxxxxxxxxxxxxx wrote:
> > From: mark gross <mgross@xxxxxxxxxxxxxxx>
> > 
> > 
> > Reviewed-by: Mark Gross <mgross@xxxxxxxxxxxxxxx>
> > Signed-off-by: Mark Gross <mgross@xxxxxxxxxxxxxxx>
> 
> My reading of submitting-patches.rst seems to indicate that
> the Reviewer and Submitter are probably not the same person.
> 
> Are you sure that you reviewed it?
> 
> 
> > ---
> >  Documentation/index.rst                  |   3 +-
> >  Documentation/vpu/index.rst              |  16 ++
> >  Documentation/vpu/vpu-stack-overview.rst | 267 +++++++++++++++++++++++
> >  3 files changed, 285 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/vpu/index.rst
> >  create mode 100644 Documentation/vpu/vpu-stack-overview.rst
> > 
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index 57719744774c..0a2cc0204e8f 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
> > @@ -1,4 +1,4 @@
> > -.. SPDX-License-Identifier: GPL-2.0
> > +.. SPDX-License-Identifier: GPL-2.0-only
> 
> That looks both inappropriate for this patch and incorrect AFAICT.
> 
> >  
> >  
> >  .. The Linux Kernel documentation master file, created by
> > @@ -137,6 +137,7 @@ needed).
> >     misc-devices/index
> >     scheduler/index
> >     mhi/index
> > +   vpu/index
> >  
> >  Architecture-agnostic documentation
> >  -----------------------------------
> > diff --git a/Documentation/vpu/index.rst b/Documentation/vpu/index.rst
> > new file mode 100644
> > index 000000000000..7e290e048910
> > --- /dev/null
> > +++ b/Documentation/vpu/index.rst
> > @@ -0,0 +1,16 @@
> > +.. SPDX-License-Identifier: GPL-2.0-only
> 
> license-rules.rst says:
> 
> 	For 'GNU General Public License (GPL) version 2 only' use:
> 	  SPDX-License-Identifier: GPL-2.0
> 
> > +
> > +============================================
> > +Vision Processor Unit Documentation
> > +============================================
> > +
> > +This documentation contains information for the Intel VPU stack.
> > +
> > +.. class:: toc-title
> > +
> > +	   Table of contents
> > +
> > +.. toctree::
> > +   :maxdepth: 2
> > +
> > +   vpu-stack-overview
> > diff --git a/Documentation/vpu/vpu-stack-overview.rst b/Documentation/vpu/vpu-stack-overview.rst
> > new file mode 100644
> > index 000000000000..53c06a7d9a52
> > --- /dev/null
> > +++ b/Documentation/vpu/vpu-stack-overview.rst
> > @@ -0,0 +1,267 @@
> > +.. SPDX-License-Identifier: GPL-2.0-only
> 
> Nope.
> 
> > +
> > +======================
> > +Intel VPU architecture
> > +======================
> > +
> > +Overview
> > +========
> > +
> > +The Intel Movidius acquisition has developed a Vision Processing Unit (VPU)
> > +roadmap of products starting with Keem Bay (KMB).  The HW configurations the
> 
> s/HW/hardware/
> 
> > +VPU can support include:
> > +
> > +1. Standalone smart camera that does local CV processing in camera
> 
> Tell us what CV is before using it.
> 
> > +2. Standalone appliance or SBC device connected to a network and tethered
> 
> Tell us what SBC is before using it. (yeah, I know)
> 
> > +   cameras doing local CV processing
> > +3. Embedded in a USB dongle or M.2 as an CV accelerator.
> > +4. Multiple VPU enabled SOC's on a PCIE card as a CV accelerator in a larger IA
> 
>                                       PCIe (?)
> 
> > +   box or server.
> > +
> > +Keem Bay is the first instance of this family of products. This document
> > +provides an architectural overview of the SW stack supporting the VPU enabled
> 
> s/SW/software/
> 
> > +products.
> > +
> > +Keem Bay (KMB) is a Computer Vision AI processing SoC based on ARM A53 CPU that
> > +provides Edge neural network acceleration (inference) and includes a Vision
> > +Processing Unit (VPU) hardware.  The ARM CPU SubSystem (CPUSS) interfaces
> > +locally to the VPU and enables integration/interfacing with a remote host over
> > +PCIe or USB or Ethernet interfaces. The interface between the CPUSS and the VPU
> > +is implemented with HW FIFOs (Control) and coherent memory mapping (Data) such
> > +that zero copy processing can happen within the VPU.
> > +
> > +The KMB can be used in all 4 of the above classes of designs.
> > +
> > +We refer to the 'local host' as being the ARM part of the SoC, while the
> > +'remote host' as the IA system hosting the KMB device(s).  The KMB SoC boots
> > +from an eMMC via uBoot and ARM Linux compatible device tree interface with an
> > +expectation to fully boot within hundreds of milliseconds.  There is also
> > +support for downloading the kernel and root file system image from a remote
> > +host.
> > +
> > +The eMMC can be updated with standard mender update process.
> 
>                                          Mender
> 
> > +See https://github.com/mendersoftware/mender
> > +
> > +The VPU is started and controlled from the A53 local host.  Its firmware image
> > +is loaded using the drive FW helper KAPI's.
> 
> s/FW/firmware/
> 
> > +
> > +The VPU IP FW payload consists of a SPARC ISA RTEMS bootloader and/or
> > +application binary.
> > +
> > +The interface allowing (remote or local) host clients  to access VPU IP
> 
>                                                         ^^drop one space
> 
> > +capabilities is realized through an abstracted programming model, which
> > +provides Remote Proxy APIs for a host CPU application to dynamically create and
> > +execute CV and NN workloads on the VPU. All frameworks exposed through
> 
> Tell us what NN is.
> 
> > +programming model’s APIs are contained in the pre-compiled standard firmware
> > +image.
> > +
> > +There is a significant SW stack built up to support KMB and the use cases.  The
> > +rest of this documentation provides an overview of the components of the stack.
> > +
> > +Keem Bay IPC
> > +============
> > +
> > +Directly interfaces with the KMB HW FIFOs to provide zero copy processing from
> > +the VPU.  It implements the lowest level protocol for interacting with the VPU.
> > +
> > +The Keem Bay IPC mechanism is based on shared memory and hardware FIFOs,
> 
>                                                                      FIFOs.
> 
> > +specifically there are:
> 
>    Specifically
> 
> > +
> > +* Two 128-entry HW FIFOs, one for the CPU and one for the VPU.
> > +* Two shared memory regions, used as memory pool for allocating IPC buffers
> 
> end with a period since the previous line did that.
> 
> > +
> > +An IPC channel is a software abstraction allowing communication multiplexing,
> > +so that multiple applications / users can concurrently communicated to the VPU.
> 
>                                                           communicate
> 
> > +IPC channels area conceptually similar to socket ports.
> > +
> > +There is a total of 1024 channels, each one identified by a channel ID, ranging
> 
>          are
> 
> > +from 0 to 1023.
> > +
> > +Channels are divided in two categories:
> > +
> > +* High-Speed (HS) channels, having IDs in the 0-9 range.
> > +* General-Purpose (GP) channels, having IDs in the 10-1023 range.
> > +
> > +HS channels have higher priority over GP channels and can be used by
> > +applications requiring higher throughput or lower latency.
> > +
> > +Since all the channels share the same HW resources (i.e., the HW FIFOs and the
> > +IPC memory pools), the Keem Bay IPC driver uses software queues to give a
> > +higher priority to HS channels.
> > +
> > +The driver supports a build-time configurable number of communication channels
> > +defined in a so called Channel Mapping Table.
> 
>                 so-called
> 
> > +
> > +An IPC channel is full duplex: a pending operation from a certain channel does
> > +not block other operations on the same channel, regardless of their operation
> > +mode (blocking or non-blocking).
> > +
> > +Operation mode is individually selectable for each channel, per operation
> > +direction (read or write). All operations for that direction comply to
> > +selection.
> > +
> > +
> > +Keem Bay-VPU-IPC
> > +================
> > +
> > +This is the MMIO driver of the VPU IP block inside the SOC. It is a control
> > +driver mapping IPC channel communication to Xlink virtual channels.
> > +
> > +This driver provides the following functionality to other drivers in the
> > +communication stack:
> > +
> > +* VPU IP execution control (firmware load, start, reset)
> > +* VPU IP event notifications (device connected, device disconnected, WDT event)
> > +* VPU IP device status query (OFF, BUSY, READY, ERROR, RECOVERY)
> > +* Communication via the IPC protocol (wrapping the Keem Bay IPC driver and
> > +  exposing it to higher level Xlink layer)
> > +
> > +In addition to the above, the driver exposes SoC information (like stepping,
> > +device ID, etc.) to user-space via sysfs.
> > +
> > +This driver depends on the 'Keem Bay IPC' driver, which enables the Keem Bay
> > +IPC communication protocol.
> > +
> > +The driver uses the Firmware API to load the VPU firmware from user-space.
> > +
> > +Xlink-IPC
> > +=========
> > +This component is implementing the IPC specific Xlink protocol. It maps channel
> 
>         component implements          IPC-specific
> 
> 
> > +IDs to HW FIFO entries, using the Keem Bay VPU IPC driver.
> > +
> > +Some of the main functions this driver provides:
> > +
> > +* establishing a connection with an IPC device
> > +* obtaining a list with the available devices
> > +* obtaining the status for a device
> > +* booting a device
> > +* resetting a device
> > +* opening and closing channels
> > +* issuing read and write operations
> > +
> > +Xlink-core
> > +==========
> > +
> > +This component implements an abstracted set of control and communication APIs
> > +based on channel identification. It is intended to support VPU technology both
> > +at SoC level as well as at IP level, over multiple interfaces.
> > +
> > +It provides symmetrical services, where the producer and the consumer have
> > +the same privileges.
> > +
> > +Xlink driver has the ability to abstract several types of communication
> > +channels underneath, allowing the usage of different interfaces with the same
> > +function calls.
> > +
> > +Xlink services are available to both kernel and user space clients and include:
> > +
> > +* interface abstract control and communication API
> > +* multi device support
> > +* concurrent communication across 4096 communication channels (from 0 to
> > +  0xFFF), with customizable properties
> > +* full duplex channels with multiprocess and multithread support
> > +* channel IDs can be mapped to desired physical interface (PCIE, USB, ETH, IPC)
> 
>                                                               PCIe
> 
> > +  via a Channel Mapping Table
> > +* asynchronous fast pass through mode: remote host data packets are directly
> 
>                        passthrough
> 
> > +  dispatched using interrupt systems running on local host to IPC calls for low
> > +  overhead
> > +* channel handshaking mechanism for peer to peer communication, without the
> > +  need of static channel preallocation
> > +* channel resource management
> > +* asynchronous data and device notifications to subscribers
> > +
> > +Xlink transports: PCIe, USB, ETH, IPCXLink-PCIe
> 
>                                      IPC,
> 
> > +
> > +XLink-PCIE
> 
>          PCIe
> 
> > +==========
> > +This is an endpoint driver that is mapping Xlink channel IDs to PCIE channels.
> 
>                               that maps                            PCIe
> 
> > +
> > +This component ensures (remote)host-to-(local)host communication, and VPU IP
> > +communication via an asynchronous pass through mode, where PCIE data loads are
> 
>                                      passthrough              PCIe
> 
> 
> > +directly dispatched to Xlink-IPC.
> > +
> > +The component builds and advertises Device IDs that can are used by local host
> 
>                                                   that are used
> 
> > +application in case of multi device scenarios.
> > +
> > +XLink-USB
> > +==========
> > +This is an endpoint driver that is mapping Xlink channel IDs to bidirectional
> 
>                               that maps
> 
> > +USB endpoints and supports CDC USB class protocol. More than one Xlink channels
> 
>                                                                           channel
> 
> > +can be mapped to a single USB endpoint.
> > +
> > +This component ensures host-to-host communication, and, as well, asynchronous
> > +pass through communication, where USB transfer packets are directly dispatched
> 
>   passthrough
> 
> > +to Xlink-IPC.
> > +
> > +The component builds and advertises Device IDs that can are used by local host
> 
>                                                   that are used
> 
> > +application in case of multi device scenarios.
> > +
> > +XLink-ETH
> > +=========
> > +
> > +This is an endpoint driver that is mapping Xlink channel IDs to Ethernet
> 
>                               that maps
> 
> > +sockets.
> > +
> > +This component ensures host-to-host communication, and, as well, asynchronous
> > +pass through communication, where Ethernet data loads are directly dispatched to
> 
>    passthrough
> 
> > +Xlink-IPC.
> > +
> > +The component builds and advertises Device IDs that can are used by local host
> 
>                                                   that are used
> 
> > +application in case of multi device scenarios.
> > +
> > +Assorted drivers that depend on this stack:
> > +
> > +Xlink-SMB
> > +=========
> > +The Intel Edge.AI Computer Vision platforms have to be monitored using platform
> > +devices like sensors, fan controller, IO expander etc. Some of these devices
> > +are memory mapped and some are i2c based. Either of these devices are not
> 
>                                   I2C-based. None of these devices is
> 
> > +directly accessible to the host.
> > +
> > +The host here refers to the server to which the vision accelerators are
> > +connected over PCIe Interface. The Host needs to do a consolidated action based
> > +on the parameters of platform devices. In general, most of the standard devices
> > +(includes sensors, fan controller, IO expander etc) are I2C/SMBus based and are
> 
>                                                                SMBus-based
> 
> > +used to provide the status of the accelerator. Standard drivers for these
> > +devices are available based on i2c/smbus APIs.
> 
>                                   I2C/SMBus
> 
> > +
> > +Instead of changing the sensor drivers to adapt to PCIe interface, a generic
> > +i2c adapter "xlink-smbus" which underneath uses xlink as physical medium is
> 
>    I2C                                             Xlink
> 
> > +used. With xlink-smbus, the drivers for the platform devices doesn't need to
> 
>                                                                 don't
> 
> > +undergo any interface change.
> > +
> > +TSEN
> > +====
> > +
> > +Thermal sensor driver for exporting thermal events to the local Arm64 host as
> > +well as to the remote X86 host if in the PCIe add in CV accelerator
> 
>                                                  add-in
> 
> > +configuration.
> > +
> > +The driver receiving the junction temperature from different heating points
> 
>               receives
> 
> > +inside the SOC. The driver will receive the temperature on SMBUS connection and
> 
>                                                               SMBus
> 
> > +forward over xlink-smb when in a remote host configuration.
> > +
> > +In Keem Bay, the four thermal junction temperature points are, Media Subsystem
> 
>                                                                 ^no comma
> 
> > +(mss), NN subsystem (nce), Compute subsystem (cse) and SOC(Maximum of mss, nce
> 
>                                                       and SOC (maximum of mss, nce
> > +and cse)
> 
>        cse).
> 
> > +
> > +HDDL
> > +====
> > +
> > +- Exports details of temperature sensor, current sensor and fan controller
> > +  present in Intel Edge.AI Computer Vision platforms to IA host.
> > +- Enable Time sync of Intel Edge.AI Computer Vision platform with IA host.
> > +- Handles device connect and disconnect events.
> > +- Receives slave address from the IA host for memory mapped thermal sensors
> > +  present in SoC (Documentation/hwmon/intel_tsens_sensors.rst).
> > +- Registers i2c slave device for slaves present in Intel Edge.AI Computer
> 
>                I2C
> 
> > +  Vision platform
> > +
> > +
> > +VPUMGR (VPU Manager)
> > +====================
> > +
> > +Bridges firmware on VPU side and applications on CPU user-space, it assists
> > +firmware on VPU side serving multiple user space application processes on CPU
> > +side concurrently while also performing necessary data buffer management on
> > +behalf of VPU IP.
> > 
> 
> 
> -- 
> ~Randy
Thanks for the great feedback I've just applied it to my tree for the next
posting.

--mark