Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
From: Wu Hao
Date: Sat Apr 01 2017 - 07:32:23 EST
On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
> On Fri, Mar 31, 2017 at 1:24 PM, <matthew.gerlach@xxxxxxxxxxxxxxx> wrote:
> >
> >
> > On Thu, 30 Mar 2017, Wu Hao wrote:
> >
> >
> > Hi Wu Hao,
> >
> > Great documentation. I'm looking forward to diving into the rest of the
> > patches. Please see my comments inline.
> >
> > Matthew Gerlach
> >
> >
> >> Add a document for Intel FPGA driver overview.
> >>
> >> Signed-off-by: Enno Luebbers <enno.luebbers@xxxxxxxxx>
> >> Signed-off-by: Xiao Guangrong <guangrong.xiao@xxxxxxxxxxxxxxx>
> >> Signed-off-by: Wu Hao <hao.wu@xxxxxxxxx>
> >> ---
> >> Documentation/fpga/intel-fpga.txt | 259
> >> ++++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 259 insertions(+)
> >> create mode 100644 Documentation/fpga/intel-fpga.txt
> >>
> >> diff --git a/Documentation/fpga/intel-fpga.txt
> >> b/Documentation/fpga/intel-fpga.txt
> >> new file mode 100644
> >> index 0000000..9396cea
> >> --- /dev/null
> >> +++ b/Documentation/fpga/intel-fpga.txt
> >> @@ -0,0 +1,259 @@
> >>
> >> +===============================================================================
> >> + Intel FPGA driver Overview
> >>
> >> +-------------------------------------------------------------------------------
> >> + Enno Luebbers <enno.luebbers@xxxxxxxxx>
> >> + Xiao Guangrong <guangrong.xiao@xxxxxxxxxxxxxxx>
> >> + Wu Hao <hao.wu@xxxxxxxxx>
> >> +
> >> +The Intel FPGA driver provides interfaces for userspace applications to
> >> +configure, enumerate, open, and access FPGA accelerators on platforms
> >> equipped
> >> +with Intel(R) FPGA solutions and enables system level management
> >> functions such
> >> +as FPGA reconfiguration, power management, and virtualization.
> >> +
> >
> >
> > From a Linux kernel perspective, I'm not sure this is the best name for
> > this code. The name gives me the impression that it is a driver for all
> > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> > PCIe bus. The processor could be directely connected like the Arria10
> > SOCFPGA. Such a processor could certainly benefit from this accelerator
> > usage model. In an extreme case, couldn't a processor in the FPGA,
> > running Linux, also benefit from this accelerator model? Is this code a
> > "FPGA Accelerator Framework"?
> >
> >> +HW Architecture
> >> +===============
> >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
> >> device.
> >> +The FPGA device memory is organized using a predefined data structure
> >> (Device
> >> +Feature List). Features supported by the particular FPGA device are
> >> exposed
> >> +through these data structures, as illustrated below:
> >> +
> >> + +-------------------------------+ +-------------+
> >> + | PF | | VF |
> >> + +-------------------------------+ +-------------+
> >> + ^ ^ ^ ^
> >> + | | | |
> >> ++-----|------------|---------|--------------|-------+
> >> +| | | | | |
> >> +| +-----+ +-------+ +-------+ +-------+ |
> >> +| | FME | | Port0 | | Port1 | | Port2 | |
> >> +| +-----+ +-------+ +-------+ +-------+ |
> >> +| ^ ^ ^ |
> >> +| | | | |
> >> +| +-------+ +------+ +-------+ |
> >> +| | AFU | | AFU | | AFU | |
> >> +| +-------+ +------+ +-------+ |
> >> +| |
> >> +| FPGA PCIe Device |
> >> ++---------------------------------------------------+
> >> +
> >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
> >> can be
> >> +used to assign individual accelerators to virtual machines .
> >
> >
> > Does this HW Architecture require an Intel FPGA? Couldn't any vendors FPGA
> > be used as long as it presented itself the PCIe bus the same and contained
> > an appropriate Device Feature List?
> >
> >> +
> >> +FME (FPGA Management Engine)
> >> +============================
> >> +The FPGA Management Enging performs power and thermal management, error
> >> +reporting, reconfiguration, performance reporting, and other
> >> infrastructure
> >> +functions. Each FPGA has one FME, which is always accessed through the
> >> physical
> >> +function (PF).
> >> +
> >> +User-space applications can acquire exclusive access to the FME using
> >> open(),
> >> +and release it using close().
> >> +
> >> +The following functions are exposed through ioctls:
> >> +
> >> + Get driver API version (FPGA_GET_API_VERSION)
> >> + Check for extensions (FPGA_CHECK_EXTENSION)
> >> + Assign port to PF (FPGA_FME_PORT_ASSIGN)
> >> + Release port from PF (FPGA_FME_PORT_RELEASE)
> >> + Program bitstream (FPGA_FME_PORT_PR)
> >> +
> >> +More functions are exposed through sysfs
> >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> >> +
> >> + Read bitstream ID (bitstream_id)
> >> + Read bitstream metadata (bitstream_metadata)
> >> + Read number of ports (ports_num)
> >> + Read socket ID (socket_id)
> >> + Read performance counters (perf/)
> >> + Power management (power_mgmt/)
> >> + Thermal management (thermal_mgmt/)
> >> + Error reporting (errors/)
> >> +
> >> +PORT
> >> +====
> >> +A port represents the interface between the static FPGA fabric (the "blue
> >> +bitstream") and a partially reconfigurable region containing an AFU (the
> >> "green
>
> Is this an fpga bridge but with added features?
Yes, I think so. As you see the fme_pr function in patch 11, related port needs
to be disabled firstly before fpga_mgr_buf_load for given accelerator.
Thanks
Hao
>
> >> +bitstream"). It controls the communication from SW to the accelerator and
> >> +exposes features such as reset and debug.
> >> +
> >> +A PCIe device may have several ports and each port can be released from
> >> PF by
> >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
> >> sriov
> >> +sysfs interface.
> >> +
> >> +AFU
> >> +===
> >> +An AFU is attached to a port and exposes a 256k MMIO region to be used
> >> for
> >> +accelerator-specific control registers.
> >> +
> >> +User-space applications can acquire exclusive access to an AFU attached
> >> to a
> >> +port by using open() on the port device node, and release it using
> >> close().
> >> +
> >> +The following functions are exposed through ioctls:
> >> +
> >> + Get driver API version (FPGA_GET_API_VERSION)
> >> + Check for extensions (FPGA_CHECK_EXTENSION)
> >> + Get port info (FPGA_PORT_GET_INFO)
> >> + Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> >> + Map DMA buffer (FPGA_PORT_DMA_MAP)
> >> + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> >> + Reset AFU (FPGA_PORT_RESET)
> >> + Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> >> + Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> >> + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> >> + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> >> +
> >> +User-space applications can also mmap() accelerator MMIO regions.
> >> +
> >> +More functions are exposed through sysfs:
> >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> >> +
> >> + Read Accelerator GUID (afu_id)
> >> + Error reporting (errors/)
> >> +
> >> +Partial Reconfiguration
> >> +=======================
> >> +As mentioned above, accelerators can be reconfigured through partial
> >> +reconfiguration of a green bitstream file (GBS). The green bitstream must
> >> have
> >> +been generated for the exact blue bitstream and targeted reconfigurable
> >> region
> >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
> >> and
> >> +possibly cause system instability. This compatibility can be checked by
> >> +comparing the interface ID noted in the GBS header against the interface
> >> ID
> >> +exposed by the FME through sysfs (see above). This check is usually done
> >> by
> >> +user-space before calling the reconfiguration IOCTL.
> >> +
> >> +FPGA virtualization
> >> +===================
> >> +To enable accessing an accelerator from applications running in a VM, the
> >> +respective AFU's port needs to be assigned to a VF using the following
> >> steps:
> >> +
> >> + a) The PF owns all AFU ports by default. Any port that needs to be
> >> reassigned
> >> + to a VF must be released from PF firstly through the
> >> FPGA_FME_PORT_RELEASE
> >> + ioctl on the FME device.
> >> +
> >> + b) Once N ports are released from PF, then user can use below command to
> >> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> >> +
> >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> >> +
> >> + c) Pass through the VFs to VMs
> >> +
> >> + d) The AFU under VF is accessiable from applications in VM (using the
> >> same
> >> + driver inside the VF).
> >> +
> >> +Note the an FME can't be assigned to a VF, thus PR and other management
> >> +functions are only available via the PF.
> >> +
> >> +
> >> +Driver organization
> >> +===================
> >> +
> >> + +------------------+ +---------+ | +---------+
> >> + | +-------+ | | | | | |
> >> + | | FPGA | FME | | AFU | | | AFU |
> >> + | |Manager| Module | | Module | | | Module |
> >> + | +-------+ | | | | | |
> >> + +------------------+ +---------+ | +---------+
> >> + +-----------------------+ | +-----------------------+
> >> + | FPGA Container Device | | | FPGA Container Device |
> >> + +-----------------------+ | +-----------------------+
> >> + +------------------+ | +------------------+
> >> + | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
> >> + +------------------+ Host | Machine +------------------+
> >> + ------------------------------------ | ------------------------------
> >> + +---------------+ | +---------------+
> >> + | PCI PF Device | | | PCI VF Device |
> >> + +---------------+ | +---------------+
> >> +
> >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
> >> device
> >> +driver is always loaded first once a FPGA PCIE PF or VF device is
> >> detected. This
> >> +driver plays an infrastructural role in the driver architecuture. It:
> >> +
> >> + a) creates FPGA container device as parent of the feature devices.
> >> + b) walks through the Device Feature List, which is implemented in
> >> PCIE
> >> + device BAR memory, to discover feature devices and their sub
> >> features
> >> + and create platform device for them under the container device.
> >
> >
> > I really like the idea of creating platform devices for the sub features. It
> > is in line with other FPGA use cases. Platform devices are at the heart of
> > device trees used by processors directly connected FPGAs and processors
> > inside FPGAs.
> >
> >> + c) supports SRIOV.
> >> + d) introduces the feature device infrastructure, which abstracts
> >> + operations for sub features and exposes common functions to
> >> feature
> >> + device drivers.
> >> +
> >> +The FPGA Management Engine (FME) driver is a platform driver which is
> >> loaded
> >> +automatically after FME platform device creation from the PCIE driver. It
> >> +provides the key features for FPGA management, including:
> >> +
> >> + a) Power and thermal management, error reporting, performance
> >> reporting
> >> + and other infrastructure functions. Users can access these
> >> functions
> >> + via sysfs interfaces exposed by FME driver.
> >> + b) Paritial Reconfiguration. The FME driver registers a FPGA
> >> Manager
> >> + during PR sub feature initialization; once it receives an
> >> + FPGA_FME_PORT_PR ioctl from user, it invokes the common
> >> interface
> >> + function from FPGA Manager to complete the partial
> >> reconfiguration of
> >> + the bitstream to the given port.
> >> + c) Port management for virtualization. The FME driver introduces
> >> two
> >> + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> >> + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
> >> port is
> >> + released from the PF, it can be assigned to the VF through the
> >> SRIOV
> >> + interfaces provided by PCIE driver. (Refer to "FPGA
> >> virtualization"
> >> + for more details).
> >> +
> >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
> >> driver
> >> +is probed once the AFU platform device is created. The main function of
> >> this
> >> +module is to provide an interface for userspace applications to access
> >> the
> >> +individual accelerators, including basic reset control on port, AFU MMIO
> >> region
> >> +export, dma buffer mapping service, UMsg notification, and remote debug
> >> +functions (see above).
> >> +
> >> +
> >> +Device enumeration
> >> +==================
> >> +This section introduces how applications enumerate the fpga device from
> >> +the sysfs hierarchy under /sys/class/fpga.
> >> +
> >> +In the example below, two Intel(R) FPGA devices are installed in the
> >> host. Each
> >> +fpga device has one FME and two ports (AFUs).
> >> +
> >> +For each FPGA device, a device director is created under
> >> /sys/class/fpga/:
> >> +
> >> + /sys/class/fpga/fpga.0
> >> + /sys/class/fpga/fpga.1
> >> +
> >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
> >> name.
> >> +Application can retrieve name information via the sysfs interface:
> >> +
> >> + /sys/class/fpga/fpga.0/name
> >> +
> >> +Each node has one FME and two ports (AFUs) as child devices:
> >> +
> >> + /sys/class/fpga/fpga.0/intel-fpga-fme.0
> >> + /sys/class/fpga/fpga.0/intel-fpga-port.0
> >> + /sys/class/fpga/fpga.0/intel-fpga-port.1
> >> +
> >> + /sys/class/fpga/fpga.1/intel-fpga-fme.1
> >> + /sys/class/fpga/fpga.1/intel-fpga-port.2
> >> + /sys/class/fpga/fpga.1/intel-fpga-port.3
> >> +
> >> +In general, the FME/AFU sysfs interfaces are named as follows:
> >> +
> >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> >> +
> >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
> >> all
> >> +ports.
> >> +
> >> +The device nodes used for ioctl() or mmap() can be referenced through:
> >> +
> >> + /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> >> + /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> >> +
> >> +
> >> +Open discussions
> >> +================
> >> +The current FME driver does not provide user space access to the FME MMIO
> >> +region, but exposes access through sysfs and ioctls. It also provides an
> >> FPGA
> >> +manger interface for partial reconfiguration (PR), but does not make use
> >> of
> >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
> >> inside
> >> +the FME, and fpga-region depends on device tree which is not used at all.
> >> There
> >> +are patches from Alan Tull to separate the device tree specific code and
> >
> >
> > I am currently trying to use those patches in a different driver. They've
> > compiled cleanly in my out of tree pcie module driver against the 3.10
> > kernel.
> > I need to actually write the code to create and register the region, but
> > Alan's platform driver code should be a good guide for me. Just need to
> > find the time.
> >
> >> +introduce a sysfs interface for PR. We plan to add fpga-regions support
> >> in the
> >> +driver once the related patches get merged. Then the FME driver should
> >> create
> >> +one fpga-region for each Port/AFU.
> >
> >
> > Does the FME driver create the fpga-region, or is each region described as
> > an entry in the Device Feature List and therefore created by the code that
> > enumerates the Device Feature List?
> >
> >> --
> >> 2.7.4
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >