Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull]

On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@xxxxxxxxxxxxxxx> wrote:
>
>
> On Thu, 30 Mar 2017, Wu Hao wrote:
>
>
> Hi Wu Hao,
>
> Great documentation. I'm looking forward to diving into the rest of the
> patches. Please see my comments inline.
>
> Matthew Gerlach
>
>
>> Add a document for Intel FPGA driver overview.
>>
>> Signed-off-by: Enno Luebbers <enno.luebbers@xxxxxxxxx>
>> Signed-off-by: Xiao Guangrong <guangrong.xiao@xxxxxxxxxxxxxxx>
>> Signed-off-by: Wu Hao <hao.wu@xxxxxxxxx>
>> ---
>> Documentation/fpga/intel-fpga.txt | 259
>> ++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 259 insertions(+)
>> create mode 100644 Documentation/fpga/intel-fpga.txt
>>
>> diff --git a/Documentation/fpga/intel-fpga.txt
>> b/Documentation/fpga/intel-fpga.txt
>> new file mode 100644
>> index 0000000..9396cea
>> --- /dev/null
>> +++ b/Documentation/fpga/intel-fpga.txt
>> @@ -0,0 +1,259 @@
>>
>> +===============================================================================
>> +                    Intel FPGA driver Overview
>>
>> +-------------------------------------------------------------------------------
>> +                Enno Luebbers <enno.luebbers@xxxxxxxxx>
>> +                Xiao Guangrong <guangrong.xiao@xxxxxxxxxxxxxxx>
>> +                Wu Hao <hao.wu@xxxxxxxxx>
>> +
>> +The Intel FPGA driver provides interfaces for userspace applications to
>> +configure, enumerate, open, and access FPGA accelerators on platforms
>> equipped
>> +with Intel(R) FPGA solutions and enables system level management
>> functions such
>> +as FPGA reconfiguration, power management, and virtualization.
>> +
>
>
> From a Linux kernel perspective, I'm not sure this is the best name for
> this code.  The name gives me the impression that it is a driver for all
> Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> PCIe bus.  The processor could be directely connected like the Arria10
> SOCFPGA.  Such a processor could certainly benefit from this accelerator
> usage model.  In an extreme case, couldn't a processor in the FPGA,
> running Linux, also benefit from this accelerator model?  Is this code a
> "FPGA Accelerator Framework"?
>
>> +HW Architecture
>> +===============
>> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
>> device.
>> +The FPGA device memory is organized using a predefined data structure
>> (Device
>> +Feature List). Features supported by the particular FPGA device are
>> exposed
>> +through these data structures, as illustrated below:
>> +
>> +  +-------------------------------+  +-------------+
>> +  |              PF               |  |     VF      |
>> +  +-------------------------------+  +-------------+
>> +      ^            ^         ^              ^
>> +      |            |         |              |
>> ++-----|------------|---------|--------------|-------+
>> +|     |            |         |              |       |
>> +|  +-----+     +-------+ +-------+      +-------+   |
>> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
>> +|  +-----+     +-------+ +-------+      +-------+   |
>> +|                  ^         ^              ^       |
>> +|                  |         |              |       |
>> +|              +-------+ +------+       +-------+   |
>> +|              |  AFU  | |  AFU |       |  AFU  |   |
>> +|              +-------+ +------+       +-------+   |
>> +|                                                   |
>> +|                 FPGA PCIe Device                  |
>> ++---------------------------------------------------+
>> +
>> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
>> can be
>> +used to assign individual accelerators to virtual machines .
>
>
> Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> be used as long as it presented itself the PCIe bus the same and contained
> an appropriate Device Feature List?
>
>> +
>> +FME (FPGA Management Engine)
>> +============================
>> +The FPGA Management Enging performs power and thermal management, error
>> +reporting, reconfiguration, performance reporting, and other
>> infrastructure
>> +functions. Each FPGA has one FME, which is always accessed through the
>> physical
>> +function (PF).
>> +
>> +User-space applications can acquire exclusive access to the FME using
>> open(),
>> +and release it using close().
>> +
>> +The following functions are exposed through ioctls:
>> +
>> +       Get driver API version (FPGA_GET_API_VERSION)
>> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
>> +       Release port from PF (FPGA_FME_PORT_RELEASE)
>> +       Program bitstream (FPGA_FME_PORT_PR)
>> +
>> +More functions are exposed through sysfs
>> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
>> +
>> +       Read bitstream ID (bitstream_id)
>> +       Read bitstream metadata (bitstream_metadata)
>> +       Read number of ports (ports_num)
>> +       Read socket ID (socket_id)
>> +       Read performance counters (perf/)
>> +       Power management (power_mgmt/)
>> +       Thermal management (thermal_mgmt/)
>> +       Error reporting (errors/)
>> +
>> +PORT
>> +====
>> +A port represents the interface between the static FPGA fabric (the "blue
>> +bitstream") and a partially reconfigurable region containing an AFU (the
>> "green

Is this an fpga bridge but with added features?

>> +bitstream"). It controls the communication from SW to the accelerator and
>> +exposes features such as reset and debug.
>> +
>> +A PCIe device may have several ports and each port can be released from
>> PF by
>> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
>> sriov
>> +sysfs interface.
>> +
>> +AFU
>> +===
>> +An AFU is attached to a port and exposes a 256k MMIO region to be used
>> for
>> +accelerator-specific control registers.
>> +
>> +User-space applications can acquire exclusive access to an AFU attached
>> to a
>> +port by using open() on the port device node, and release it using
>> close().
>> +
>> +The following functions are exposed through ioctls:
>> +
>> +       Get driver API version (FPGA_GET_API_VERSION)
>> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> +       Get port info (FPGA_PORT_GET_INFO)
>> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
>> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
>> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
>> +       Reset AFU (FPGA_PORT_RESET)
>> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
>> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
>> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
>> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
>> +
>> +User-space applications can also mmap() accelerator MMIO regions.
>> +
>> +More functions are exposed through sysfs:
>> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
>> +
>> +       Read Accelerator GUID (afu_id)
>> +       Error reporting (errors/)
>> +
>> +Partial Reconfiguration
>> +=======================
>> +As mentioned above, accelerators can be reconfigured through partial
>> +reconfiguration of a green bitstream file (GBS). The green bitstream must
>> have
>> +been generated for the exact blue bitstream and targeted reconfigurable
>> region
>> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
>> and
>> +possibly cause system instability. This compatibility can be checked by
>> +comparing the interface ID noted in the GBS header against the interface
>> ID
>> +exposed by the FME through sysfs (see above). This check is usually done
>> by
>> +user-space before calling the reconfiguration IOCTL.
>> +
>> +FPGA virtualization
>> +===================
>> +To enable accessing an accelerator from applications running in a VM, the
>> +respective AFU's port needs to be assigned to a VF using the following
>> steps:
>> +
>> + a) The PF owns all AFU ports by default. Any port that needs to be
>> reassigned
>> + to a VF must be released from PF firstly through the
>> FPGA_FME_PORT_RELEASE
>> + ioctl on the FME device.
>> +
>> + b) Once N ports are released from PF, then user can use below command to
>> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
>> +
>> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
>> +
>> + c) Pass through the VFs to VMs
>> +
>> + d) The AFU under VF is accessiable from applications in VM (using the
>> same
>> + driver inside the VF).
>> +
>> +Note the an FME can't be assigned to a VF, thus PR and other management
>> +functions are only available via the PF.
>> +
>> +
>> +Driver organization
>> +===================
>> +
>> +  +------------------+  +---------+   |             +---------+
>> +  | +-------+        |  |         |   |             |         |
>> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
>> +  | |Manager| Module |  |  Module |   |             |  Module |
>> +  | +-------+        |  |         |   |             |         |
>> +  +------------------+  +---------+   |             +---------+
>> +        +-----------------------+     |      +-----------------------+
>> +        | FPGA Container Device |     |      | FPGA Container Device |
>> +        +-----------------------+     |      +-----------------------+
>> +          +------------------+        |         +------------------+
>> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
>> +          +------------------+   Host | Machine +------------------+
>> + ------------------------------------ | ------------------------------
>> +           +---------------+          |          +---------------+
>> +           | PCI PF Device |          |          | PCI VF Device |
>> +           +---------------+          |          +---------------+
>> +
>> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
>> device
>> +driver is always loaded first once a FPGA PCIE PF or VF device is
>> detected. This
>> +driver plays an infrastructural role in the driver architecuture.  It:
>> +
>> +       a) creates FPGA container device as parent of the feature devices.
>> +       b) walks through the Device Feature List, which is implemented in
>> PCIE
>> +          device BAR memory, to discover feature devices and their sub
>> features
>> +          and create platform device for them under the container device.
>
>
> I really like the idea of creating platform devices for the sub features. It
> is in line with other FPGA use cases.  Platform devices are at the heart of
> device trees used by processors directly connected FPGAs and processors
> inside FPGAs.
>
>> +       c) supports SRIOV.
>> +       d) introduces the feature device infrastructure, which abstracts
>> +          operations for sub features and exposes common functions to
>> feature
>> +          device drivers.
>> +
>> +The FPGA Management Engine (FME) driver is a platform driver which is
>> loaded
>> +automatically after FME platform device creation from the PCIE driver. It
>> +provides the key features for FPGA management, including:
>> +
>> +       a) Power and thermal management, error reporting, performance
>> reporting
>> +          and other infrastructure functions. Users can access these
>> functions
>> +          via sysfs interfaces exposed by FME driver.
>> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
>> Manager
>> +          during PR sub feature initialization; once it receives an
>> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
>> interface
>> +          function from FPGA Manager to complete the partial
>> reconfiguration of
>> +          the bitstream to the given port.
>> +       c) Port management for virtualization. The FME driver introduces
>> two
>> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
>> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
>> port is
>> +          released from the PF, it can be assigned to the VF through the
>> SRIOV
>> +          interfaces provided by PCIE driver. (Refer to "FPGA
>> virtualization"
>> +          for more details).
>> +
>> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
>> driver
>> +is probed once the AFU platform device is created. The main function of
>> this
>> +module is to provide an interface for userspace applications to access
>> the
>> +individual accelerators, including basic reset control on port, AFU MMIO
>> region
>> +export, dma buffer mapping service, UMsg notification, and remote debug
>> +functions (see above).
>> +
>> +
>> +Device enumeration
>> +==================
>> +This section introduces how applications enumerate the fpga device from
>> +the sysfs hierarchy under /sys/class/fpga.
>> +
>> +In the example below, two Intel(R) FPGA devices are installed in the
>> host. Each
>> +fpga device has one FME and two ports (AFUs).
>> +
>> +For each FPGA device, a device director is created under
>> /sys/class/fpga/:
>> +
>> +       /sys/class/fpga/fpga.0
>> +       /sys/class/fpga/fpga.1
>> +
>> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
>> name.
>> +Application can retrieve name information via the sysfs interface:
>> +
>> +       /sys/class/fpga/fpga.0/name
>> +
>> +Each node has one FME and two ports (AFUs) as child devices:
>> +
>> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
>> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
>> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
>> +
>> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
>> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
>> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
>> +
>> +In general, the FME/AFU sysfs interfaces are named as follows:
>> +
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
>> +
>> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
>> all
>> +ports.
>> +
>> +The device nodes used for ioctl() or mmap() can be referenced through:
>> +
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
>> +
>> +
>> +Open discussions
>> +================
>> +The current FME driver does not provide user space access to the FME MMIO
>> +region, but exposes access through sysfs and ioctls. It also provides an
>> FPGA
>> +manger interface for partial reconfiguration (PR), but does not make use
>> of
>> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
>> inside
>> +the FME, and fpga-region depends on device tree which is not used at all.
>> There
>> +are patches from Alan Tull to separate the device tree specific code and
>
>
> I am currently trying to use those patches in a different driver.  They've
> compiled cleanly in my out of tree pcie module driver against the 3.10
> kernel.
> I need to actually write the code to create and register the region, but
> Alan's platform driver code should be a good guide for me.  Just need to
> find the time.
>
>> +introduce a sysfs interface for PR. We plan to add fpga-regions support
>> in the
>> +driver once the related patches get merged. Then the FME driver should
>> create
>> +one fpga-region for each Port/AFU.
>
>
> Does the FME driver create the fpga-region, or is each region described as
> an entry in the Device Feature List and therefore created by the code that
> enumerates the Device Feature List?
>
>> --
>> 2.7.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>